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Forward 


In  August  1995, 1  commissioned  a  one  year  study  with  the  objective  of  assessing 
the  effectiveness  of  the  use  of  modeling  and  simulation  (M&S)  in  the  acquisition 
process.  In  particular,  I  was  looking  for  the  metrics  by  which  the  Department  of 
Defense  could  ascertain  the  value — if  any — that  was  returned  on  its  investment  in 
M&S  in  terms  of  the  reduction  in  time,  resources,  and  risk  in  weapons  systems 
development  and  fielding  and  in  terms  of  increase  in  the  military  utility  of  those 
systems.  I  also  tasked  the  study  team  while  gathering  information  to  support  the 
assessment  to  note  technical  and  other  challenges  to  realizing  the  postulated  benefits 
and  to  report  on  specific  M&S  tools  and  processes  being  used  to  facilitate  the 
acquisition  of  systems  in  the  DoD  and  industry. 

This  report,  which  documents  the  results  of  that  study,  provides  tangible, 
quantitative  indicators  that  the  use  of  M&S  can  provide  substantial  benefit  measured 
in  time,  cost,  productivity,  and  system  quality  and  performance.  The  evidence  is 
consistent  and  pervasive,  across  both  DoD  and  industry.  I  personally  was  impressed 
that  the  most  significant  return  on  investment  was  realized  when  M&S  was  used  as  an 
integrator  of  functions  within  the  acquisition  process,  i.e.,  integrating  design  and 
manufacturing  or  linking  requirements  more  closely  to  test.  This  leads  me  to  believe 
that  its  real  value  lies  as  an  enabler  of  Integrated  Product  and  Process  Development 
(IPPD). 

The  research  also  sheds  light  on  the  work  that  remains  to  be  accomplished  if 
the  Department  is  to  truly  realize  the  full  potential  of  M&S  in  providing  enhanced 
capability  to  its  warfighters.  Technical,  managerial,  and  cultural  challenges  remain  to 
be  addressed.  The  study  team  has  made  some  recommendations.  The  acquisition 
community  may  draw  additional  conclusions  from  the  study  on  actions  that  should  be 
taken. 


I  am  therefore  providing  this  report  on  the  ‘^Effectiveness  of  Modeling  and 
Simulation  in  the  Weapon  System  Acquisition  Process”  to  the  defense  acquisition 
community  for  the  multiple  purposes  of  alerting  acquisition  professionals  to  the 
potential  of  M&S  technologies;  of  establishing  a  broader  basis  for  supporting 
investment  and  employment  decisions  on  M&S  in  the  Department;  and  of  embarking 
on  a  strategy  to  more  fully  achieve  the  vision  of  Simulation  Based  Acquisition. 


Dr.  Patricia  Sanders 
Deputy  Director,  Test,  Systems 
Engineering  and  Evaluation 
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I. 


EXECUTIVE  SUMMARY 


A.  Background 

Dr.  Patricia  Sanders,  Deputy  Director,  Test  Facilities  and  Resources,  Office  of  the 
Director,  Test,  Systems  Engineering  and  Evaluation  (DTSE&E),  tasked  Science  Applications 
International  Corporation  (SAIC)  to  conduct  a  study  to  assess  the  effectiveness  of  using 
modeling  and  simulation  (M&S)  in  the  weapon  system  acquisition  process.  The  study  team  was 
to  investigate  metrics  that  are  being  used  to  evaluate  M&S  effectiveness;  specific  tools  that  are 
being  used  by  government  and  industry  to  facilitate  the  design,  development,  test,  manufacture, 
and  support  of  weapon  systems  in  an  Integrated  Product  and  Process  Development  (IPPD) 
environment;  the  associated  benefits  of  using  M&S  in  the  acquisition  environment;  and  technical 
challenges  that  preclude  the  seamless  use  of  M&S  in  the  acquisition  process. 

In  order  to  obtain  focused  information  on  the  issues,  the  study  team  visited 
knowledgeable  persons  in  both  Government  and  Industry.  The  focus  of  these  visits  intentionally 
excluded  training  and  requirements  definition.  Though  both  areas  benefit  from  the  use  of 
modeling  and  simulation,  studying  them  would  have  broadened  the  scope  of  the  study  and 
limited  the  time  and  resources  spent  on  other  areas  from  which  data  were  gathered. 

This  study  addresses  the  events  that  pushed  the  use  of  M&S  in  acquisition  and  what  the 
resulting  efforts  were  in  the  DoD  community.  It  also  addresses  where  this  is  taking  acquisition 
in  terms  of  opportunities  and  general  trends.  A  brief  synopsis  of  the  study  methodology  is 
presented,  along  with  examples  of  key  visits  and  efforts  noted.  Some  of  the  research  results  are 
cited,  both  in  terms  of  some  of  the  benefits  and  in  many  of  the  challenges  facing  the  DoD 
acquisition  community  today.  Finally,  findings  and  recommendations  are  presented  along  with 
summary  observations. 

The  environment  was  key  to  using  M&S  in  acquisition  processes:  M&S  tools  and 
processes  were  more  powerful  and  less  expensive  due  to  advances  in  technology;  and  declining 
resources  made  it  essential  that  better  ways  be  found  to  field  weapon  systems.  The  result  was 
that  many  weapon  system  acquisition  programs  began  using  M&S  tools  in  pieces  of  their 
acquisition  process.  At  the  same  time,  research  and  development  centers  and  test  and  evaluation 
facilities  were  developing  M&S  tools  and  processes  for  both  program  specific  and  general  use.  It 
should  be  noted  that  these  were  independent  efforts  not  coordinated  by  any  specific  agency  or 
office. 


Recent  changes  in  guidance  from  both  OSD  and  the  Services  have  begun  to  influence  the 
effort  to  manage  the  integration  of  M&S  efforts  both  within  the  programs  and  across  each 
Service.  OSD  guidance  on  using  Integrated  Product  Teams  is  impacting  the  sharing  of 
information  on  models  and  data  within  programs.  Each  of  the  Services  have  increased  their 
emphasis  on  the  use  of  M&S  in  acquisition  and  are  modifying  the  way  they  do  business. 


B.  Vision 


Although  it  is  not  clear  where  these  efforts  will  lead  the  acquisition  conununity,  it  is 
evident  that  a  significant  change  is  underway  and  that  the  end  result  will  he  a  new  way  of  doing 
business.  We  will  call  this  new  approach  to  acquisition,  “Simulation  Based  Acquisition,”  in 
order  to  differentiate  it  from  the  traditional  approaches  and  to  emphasize  its  reliance  on  the  tools 
and  processes  made  possible  by  advances  in  simulation  technology.  It  is  not  presented  here  as  a 
specific,  definable  concept,  but  rather  as  a  term  to  characterize  the  general  approach  of 
significantly  increased  and  innovative  use  of  M&S  tools  and  the  processes  which  they  enable;  a 
more  integrated  approach  to  system  development. 

Simulation  Based  Acquisition  is  the  process  by  which  simulation  is  incorporated  and 
integrated  throughout  the  fimctions  of  the  acquisition  of  a  weapon  system;  from  concept 
exploration,  through  prototyping  and  design,  test  and  evaluation,  fabrication  and  production,  to 
deployment  and  finally  operations  and  sustainment.  Using  M&S  in  acquisition  is  not  a  new  idea. 
Simulation  is  already  being  successfully  incorporated  in  each  function  in  individual  weapon 
systems  acquisition  programs  in  each  of  the  Services.  What  is  new  in  Simulation  Based 
Acquisition  is  the  integration  of  technologies  across  functions,  phases,  and  programs. 

C.  Study  Methodology 

This  report  is  the  result  of  a  one  year  study  effort  with  extensive  research  in  DoD  and 
Industry.  The  investigation  into  M&S  issues  in  acquisition  was  comprehensive  but  not 
exhaustive.  The  study  team  sought  to  provide  examples  and  metrics  of  M&S  effectiveness  in 
simulation  based  design,  virtual  prototyping,  computer  aided  engineering  and  manufacturing,  and 
simulations  used  in  testing,  support,  and  system  integration. 

Visits  were  made  to  individuals  on  the  Service  Staffs,  Research  and  Development 
Centers,  Test  and  Evaluation  agencies.  Program  Managers  and  Program  Executive  Offices,  and 
some  Industry  activities  (both  defense  and  commercial).  In  addition  to  key  individuals  on  the 
Army,  Navy,  and  Air  Staffs,  the  study  team  visited  Ford  Motor  Company,  Sikorsky  Helicopter, 
General  Motors-Truck  and  Bus  Division,  and  researched  other  companies  using  simulation  in  an 
integrated  manner  in  their  design,  development,  and  engineering  and  manufacturing  processes. 

Initial  research  gathered  from  DoD  and  Service  staffs,  as  well  as  literature  research,  led 
the  study  team  to  believe  that  Simulation  Based  Acquisition  would  show  payoffs  in  all  areas  of 
the  acquisition  process:  in  design  and  development,  where  three  dimensional  solid  models 
would  allow  rapid  evaluation  of  concept  design  and  reduction  of  physical  prototypes;  in  test, 
where  data  gathered  from  simulations  could  allow  assessment  of  what  would  not  be  easily 
testable  and  could  help  avoid  costly  repetitions  of  shots;  in  manufacturing,  to  reduce 
cost/schedule  risk  and  develop  and  validate  3-D  manufacturing  planning;  in  support  and 
integration,  to  integrate  maintenance  and  logistics  requirements  and  evaluate  new  processes. 
These  assumptions  then  provided  a  framework  for  further  research  effort. 


D.  Research  Results 


The  study  team  found  pervasive  evidence  of  M&S  being  utilized  efficiently  in  every 
Service,  though  not  in  the  same  way  and  not  yet  seamlessly  throughout  a  program.  Both 
Government  and  Industry  are  identifying  and  developing  tools  that  bring  added  benefit  to  their 
program’s  development.  Many  examples  of  individual  successes  were  found  which  were  then 
used  to  identify  metrics  to  validate  the  effective  use  of  M&S. 

Documented  successes  and  experiences  can  be  categorized  into  four  areas  of  realized 
benefits;  cost,  schedule,  productivity,  and  quality/performance.  These  categories  are  not 
mutually  exclusive.  Savings  in  productivity  can  also  be  classified  as  schedule  benefits.  The 
point  was  to  identify  benefits  cited  by  activities  and  weapon  system  acquisition  programs  and 
find  associated  metrics.  The  benefits  presented  in  this  report  are  a  fraction  of  those  encountered 
but  give  a  broad  look  at  the  types  of  examples  of  M&S  being  used  effectively  in  acquisition. 

Cost  savings  are  especially  difficult  to  quantify  and  reported  cost  savings  are  often 
illusionary.  The  cited  benefits  are  more  correctly  classified  as  “cost  avoidance”  and  are  measures 
of  significant  additional  work  or  results  that  were  obtained  using  M&S  tools  which  would  have 
cost  the  reported  “savings”  if  they  had  been  obtained  by  more  traditional  methods.  This  study 
has  been  careful  to  differentiate  between  cost  savings  and  cost  avoidance  and  to  clearly  identify 
the  latter  category  when  reported.  In  some  cases,  the  program  itself  would  not  be  economically 
feasible  without  these  improved  methods  which  provide  adequate  risk  reduction  for  an  affordable 
cost. 


One  example  of  expected  cost  savings  based  on  a  six  month  side  by  side  comparison  is 
given  by  the  Joint  Strike  Fighter  Program  Office.  The  study  projected  that  the  benefits  of  virtual 
manufacturing  offer  a  potential  savings  of  up  to  3%  of  life  cycle  costs  which  could  equate  to  $5 
billion. 


Two  types  of  cost  benefits  are  significant  to  note.  The  first  is  the  effort  and  resources  not 
used  in  programs  that  were  terminated  during  concept  exploration  as  a  result  of  assessment  using 
M&S  tools.  The  other  is  programs,  such  as  the  Longbow  Hellfire  missile,  that  were  able  to  find 
less  expensive  approaches  using  M&S,  in  this  case  to  temper  costly  fly-to-buy  testing.  The 
Longbow  Hellfire  program  invested  in  an  anechoic  chamber  facility  that  enabled  the  program  to 
cut  back  on  the  number  of  live  missiles  fired  for  lot  verification  of  stockpiled  munitions  when 
compared  to  previous  tactical  missile  programs.  Without  the  use  of  M&S  tools,  the  Hellfire 
missile  would  have  been  unaffordable  in  life  cycle  costs. 

Schedule  benefits  were  more  readily  cited.  Process  time  savings  were  realized  in  time 
required  to  design,  analyze,  tool,  and  manufacture.  Product  cycle  time  is  a  significant  metric. 

For  example,  the  US  automobile  industry  (Big  Three)  reported  that  with  use  of  M&S  tools  and 
processes,  they  have  reduced  the  cycle  time  (from  concept  approval  to  production)  from  five 
years  down  to  three  years  and  expect  significant  further  reductions.  Electric  Boat  has  reported 
reducing  the  cycle  time  for  submarine  development  from  14  years  down  to  7  years. 


Productivity  metrics  from  M&S  use  were  easily  noted:  reduced  effort  in  terms  of  man- 
years  and  number  of  workers;  reduced  number  of  intermediate  steps,  (i.e.  mockups,  redesign, 
engineering  changes);  reduced  scrap;  and  reductions  in  manufacturing  floor  space.  For  example, 
the  working  drawings  of  the  CH-53E  Super  Stallion  aircraft’s  outside  contours  required  38 
Sikorsky  draftsmen  approximately  6  months.  The  same  task  on  the  Comanche  helicopter 
program  required  only  one  month’s  effort  by  one  engineer  using  M&S. 

Benefit  examples  in  the  quality/performance  area  revealed  several  metrics;  proper 
assembly  (good  fit),  reduced  rework,  reduced  parts  count,  and  early  design  evaluation  prior  to 
further  design  effort.  Use  of  integrated  Computer-Aided  Design  (CAD)  systems  by  Northrop  led 
to  a  first-time,  error-free  physical  mock-up  of  many  B-2  sections.  Use  of  CAD  also  assisted  in 
achieving  first-time  correct  tube  bends  for  expensive  titanium  electronic  cable  conduits. 

While  the  primary  objective  of  this  study  was  to  quantify  the  value  of  M&S  in  the  DoD 
acquisition  process,  it  was  also  recognized  that  the  same  research  would  surface  data  on 
challenges  and  opportunities  that  organizations  and  programs  were  encountering  in  effectively 
exploiting  M&S  tools  within  that  process.  Again,  this  information  is  representative  of  what  the 
team  encountered  and  not  an  all-encompassing  effort  to  gather  information  on  every  acquisition 
program. 

The  challenges  identified  during  the  course  of  this  research  have  been  organized  into 
three  groups:  technical,  cultural,  and  managerial.  While  this  grouping  and  their  labels  are 
somewhat  arbitrary,  and  some  challenges  could  be  classified  into  more  than  one  group,  it 
emphasizes  the  fact  that  not  all  of  the  problems  encountered  in  effectively  using  M&S  tools  in 
the  acquisition  process  are  technical  in  nature.  A  list  of  the  challenges  described  in  this  report 
are  presented  in  Table  I-l. 

As  expected,  some  of  the  frequently  encountered  technical  problems  are  interoperability 
of  M&S  tools,  availability  of  data  descriptions,  security  of  data,  and  hardware  and  software 
limitations.  Examples  of  each  of  these  are  given  in  Chapter  IV.  The  technical  challenges 
identified  are  not  beyond  the  capability  of  improving  technology  to  resolve,  although  prudent 
investments  or  proper  encouragement  might  well  speed  and  enhance  the  usefulness  of  the 
technical  solutions. 

Incentives  for  M&S  use,  availability  of  trained  personnel,  and  acceptance  of  M&S  tools 
and  processes  are  among  the  cultural  challenges  the  study  team  encountered  while  conducting 
this  study.  Resistance  to  change  from  an  individual-based  to  a  team-based  approach  and 
retaining  technically  proficient  personnel  are  difficulties  the  acquisition  community  must 
overcome. 

Proprietary  data  and  model  issues  and  resistance  to  funding  verification  and  validation 
requirements  are  two  common  problems  that  the  team  perceived  as  managerial  challenges.  There 
is  also  a  perception  of  inadequate  guidance  or  direction  for  investing  in  M&S  tools  and 
processes. 


Table  I-l .  Challenges  for  M&S  in  the  Acquisition  Process. 


Type 

Challenges 

Technical 

•  Interoperability  of  M&S  Tools 

•  Availability  of  Data  Descriptions 

•  Security/Sensitivity  of  Data 

•  Physics-based  M&S 

•  Hardware  and  Software  Limitations 

•  Variable  Resolution 

Cultural 

•  Acquisition  Processes 

•  Incentives  for  M&S  Use 

•  M&S  Workforce 

•  Acceptance  of  M&S 

Managerial 

•  OSD  and  Service  Guidance 

•  Ownership  of  Data 

•  W&A  Requirements 

•  Funding  Process 

•  Use  of  System  Models 

E.  Observations  and  Recommendations 

There  is  real  evidence  of  M&S  tools  and  processes  being  used  effectively  and  efficiently 
in  select  programs  and  activities  in  each  of  the  services.  Examples  of  early  and  continuous 
interaction  between  users,  developers,  and  testers  are  evident  in  several  programs.  Use  of  M&S 
and  virtual  environments  to  quickly  examine  the  impact  and  results  of  decisions  have  formed  the 
basis  for  better  decision  making.  Risk  is  better  managed  through  the  analysis  of  virtual  mockups 
in  virtual  environments.  There  are  examples  of  reduction  in  cycle  time  and  greater  flexibility  in 
decision  making  through  the  use  of  M&S. 

Industry  and  Government  are  cooperating  closely  in  many  areas  pertaining  to  M&S  and 
streamlined  acquisition.  In  reaction  to  global  or  national  competition  and  other  pressures,  many 
firms  have  modified  their  approach  to  product  development,  manufacturing,  and  support.  Many 
of  these  firms,  using  M&S  tools  and  processes  extensively,  have  influenced  the  revolution  in 
DoD  acquisition,  especially  firms  that  are  DoD  contractors,  and  they  are  forging  eloser  alliances 
through  teamwork  and  IPT/IPPD  initiatives.  Most  large  system  contractors  cannot  afford 
mistakes  during  development,  thus  rely  increasingly  on  the  capabilities  provided  by  M&S.  The 
result  is  an  increase  in  number  and  influence  of  various  consortia.  Cooperative  Research  and 
Development  Agreements  (CRD As),  and  dual-use  efforts.  These  trends  lead  to  greater 
cooperation  and  efficiency.  The  opportunities  in  which  DoD  can  leverage  current  efforts  are 
tremendous.  A  number  of  the  ongoing  programs  which  provide  innovative  approaches  are 
summarized  in  Chapter  IV. 


One  key  effort  is  DARPA’s  Simulation  Based  Design  (SBD)  program.  The  goal  of  the 
project  is  to  “revolutionize  the  Acquisition  Process  for  complex  military  and  commercial 
products”  using  distributed,  collaborative  virtual  development  environments.  SBD  considers  all 
aspects  of  the  system  acquisition  process,  from  mission  analysis,  through  design  and  logistics 
considerations,  to  manufacturing  and  cost/risk  analysis  phases.  The  underlying  premise  is  that 
M&S  can  reduce  time  and  cost  for  all  areas  of  the  acquisition  process. 

Recommendation:  Opportunities  to  cooperate  with  Industry  such  as  the  DARPA 
Simulation  Based  Design  programs  should  be  encouraged  and  continued.  There  appears  to  be 
great  potential  in  partnerships  such  as  the  National  Automotive  Center  where  both  the 
Government  and  Industry  benefit  from  investigating  new  technology.  There  should  he  incentives 
to  pursue  business  relationships  such  as  these  in  order  to  utilize  developing  technology  more 
efficiently. 

A  frequently  discussed  topic  in  the  study  team’s  visits  was  the  need  for  investment  early 
in  the  program  for  the  tools  that  would  be  useful  throughout  the  weapon  system’s  life  cycle.  For 
programs  such  as  the  F-22  and  the  Comanche,  there  was  no  choice  but  to  plan  to  invest  early  in 
M&S  and  plan  their  program  around  those  investments.  For  many  smaller  programs  the  decision 
is  more  difficult.  Guidance  is  general  and  there  is  little  incentive  for  program  managers  to 
commit  their  early  program  funds  to  technologies  for  which  they  don’t  see  an  immediate  return. 
This  is  complicated  by  the  fact  that  there  are  no  funding  lines  specifically  designated  for 
investing  in  the  simulations  necessary  to  support  the  acquisition  process. 

Investment  in  simulation  facilities  extends  beyond  the  money  invested  by  weapon  system 
programs.  The  anechoic  chambers  at  Edwards  Air  Force  Base  and  Patuxent  River  Naval  Air 
Station  and  the  wind  tunnel  facility  and  computational  fluid  dynamics  capabilities  at  Arnold 
Engineering  Development  Center  are  examples  of  these  facilities.  The  Simulation  Test 
Acceptance  Facility  (STAF)  at  Redstone  Technical  Test  Center  was  funded  by  program 
managers,  but  is  a  facility  that  could  be  used  with  minor  modification  by  other  missile  programs. 

Recommendation:  To  meet  the  challenge  of  institutionalizing  the  use  of  available  M&S 
technology,  the  Services  must  be  committed  to  providing  funds  for  M&S  at  the  inception  of  the 
program.  OSD  and  the  Services  should  commit  Science  and  Technology  dollars  to  upgrade 
capabilities  and  facilities  that  could  serve  many  weapon  system  acquisitions,  and  program 
managers  should  be  encouraged  to  use  these  facilities  instead  of  contracting  to  have  new  system 
specific  facilities  built. 

There  is  currently  no  vehicle  to  get  information  on  M&S  capabilities  and  facilities  to  the 
programs  that  have  the  potential  to  utilize  the  assets.  It  would  be  useful  to  the  program  managers 
and  to  the  Research  and  Development  and  Test  and  Evaluation  activities  to  have  a  source  for 
investigating  M&S  capabilities  available  within  the  Defense  Acquisition  community. 

Recommendation:  Develop  an  information  source  such  as  an  Internet  web  page  which 
would  list  capabilities  in  design;  tools  available,  programs  that  have  used  them,  activities  they 
can  contact  for  further  information,  etc.  The  same  capabilities  could  be  listed  for  testing;  e.g. 


who  does  computational  fluid  dynamics,  which  facilities  have  wind  tunnels  and  anechoic 
chambers,  have  any  programs  proven  out  this  technology  with  their  systems?  The  web  page 
could  also  be  used  to  identify  innovative  approaches  in  manufacturing  and  note  those  using 
virtual  manufacturing  environments. 

The  words  are  in  place  in  DoD  acquisition  documents  to  support  implementation  of 
Simulation  Based  Acquisition,  though  there  are  some  growing  pains  associated  with 
implementation  of  this  shift.  For  example,  the  Multiple  Launch  Rocket  System  program  office 
submitted  an  Extended  Range  -  MLRS  Test  and  Evaluation  Master  Plan  (TEMP),  approved 
through  the  Army  Staff,  that  was  heavily  supported  by  the  results  of  simulation.  The  Office  of 
the  Director,  Operational  Test  and  Evaluation  rejected  the  TEMP  and  directed  that  36  more  live 
rockets  be  fired  than  in  the  submitted  plan.  The  issue  is,  to  what  degree  can  M&S  replace  or 
augment  field  tests?  There  is  no  universal  answer,  but  the  message  received  in  part  by  the 
community  is  that  the  thought  and  guidance  are  there,  but  the  implementation  and  acceptance  are 
not. 

Recommendation:  DoD  needs  to  institutionalize  the  use  of  models  and  simulations  and 
insure  that  the  community  is  knowledgeable  about  the  tools  available.  The  Services  and  OSD 
need  to  provide  more  responsive  guidance  relative  to  the  advent  of  better  and  more  useful 
simulation  tools.  Dialogue  is  needed  within  the  Services  and  between  the  Services  and  OSD  to 
effect  policy  on  standardization.  Program  managers  must  overcome  the  management  and 
cultural  challenges  that  present  barriers  to  the  effective  use  of  available  technology. 

The  study  found  that  program  managers  and  their  staffs  are  not  well  informed  on  M&S 
tools  and  their  use  in  acquisition.  Many  progr2un  managers  have  had  very  limited  exposure  to 
models  and  simulations  and  would  benefit  from  a  short  block  of  instruction  which  would 
emphasize  the  successes  other  programs  have  experienced  by  using  M&S.  Defense  Acquisition 
University  has  begun  to  alter  the  curriculum  in  order  to  make  the  students  more  aware  of  the 
impact  of  using  current  technology  early  in  the  acquisition  process  and  throughout  the  life  cycle 
of  the  program. 

Recommendation:  Action  is  needed  to  provide  focused  information  on  the  availability 
and  capabilities  of  M&S  to  weapon  system  acquisition  managers. 

F.  Summary 

There  is  consistent  evidence  of  M&S  being  used  effectively  in  the  acquisition  process  but 
not  in  an  integrated  manner  across  programs  or  functions  within  the  acquisition  process. 
Substantial  evidence  has  been  collected  from  individual  success  stories,  though  the  benefits  are 
not  readily  quantifiable  into  a  general  standard.  The  key  is  in  focusing  on  the  integration  of 
M&S  applications,  across  acquisition  programs  and  throughout  the  process,  not  in  exploring  the 
applications  themselves.  In  the  final  analysis,  M&S  will  continue  to  grow  in  usage,  capability 
and  total  contribution  to  the  acquisition  process.  The  opportunity  exists  now  for  the  DoD  to 
effect  Simulation  Based  Acquisition  by  focusing  on  providing  the  opportunities  to  facilitate  the 
change. 


11.  BACKGROUND 


A.  Objective  and  Approach 

1 .  Objective  and  Key  Issues 

The  objective  of  this  study  was  to  assess  the  effectiveness  of  the  use  of  modeling 
and  simulation  (M&S)  in  the  weapons  system  acquisition  and  support  processes.  These 
support  processes  include  the  systems  engineering  process,  and  evolving  processes  such 
as  concurrent  engineering  (CE)  and  Integrated  Product  and  Process  Development  (IPPD). 
The  assessment  addresses  the  following  key  questions: 

•  What  are  the  metrics  that  can  be  and  are  being  used  to  evaluate  M&S 
effectiveness  in  the  acquisition  process? 

•  What  specific  M&S  tools  are  emerging  that  are  being  used  by  Government  and 
Industry  to  facilitate  the  design,  development,  manufacturing,  test,  and  support  of 
weapons  systems  in  an  IPPD  environment? 

•  What  are  the  associated  benefits  of  using  M&S  in  the  acquisition  environment? 

•  What  are  the  key  M&S  challenges  that  remain  that  preclude  the  use  of  M&S  in  a 
seamless  way  within  the  acquisition  process? 

2.  Approach 

Recent  and  focused  information  on  policy  guidance  and  direction  was  obtained 
through  a  series  of  visits.  Important  M&S  facilities  and  efforts  were  prioritized  for 
potential  visits  in  order  to  gather  the  best  possible  information  on  the  use  of  M&S  by 
field  agencies.  A  list  of  the  key  offices  and  agencies  visited  follows  in  section  A.3  of  this 
chapter. 

Issues  important  to  the  study  were  framed  in  the  form  of  a  focus  paper  which 
presented  the  desired  questions  in  a  flexible  format.  The  goal  was  to  avoid  use  of  a 
survey  or  ‘fill  in  the  blanks’  document  in  order  to  identify  and  come  to  terms  with  key 
substantive  issues.  The  focus  paper  was  normally  tailored  for  each  organization  visited 
and  was  updated  over  the  course  of  the  study  to  include  or  highlight  key  issues.  Previous 
assessment  efforts  and  documents  facilitated  development  of  the  focus  paper.  For 
example,  the  Acquisition  Task  Force  on  M&S  (ATFMS)  had  placed  significant  emphasis 
on  developing  appropriate  questions  to  be  used  during  visits  to  a  series  of  similar 
organizations.  Such  questions  were  used  in  order  to  capture  issues  key  to  the  study’s 
objectives. 


The  study  team  also  conducted  an  extensive  literature  search  on  M&S  initiatives 
and  activities.  This  was  conducted  using  local  libraries,  including  the  Pentagon’s  Army 
library,  Defense  Systems  Management  College  (DSMC)  library,  SAIC’s  Corporate 
Technical  Resource  Acquisition  Center,  public  libraries,  the  World  Wide  Web  (WWW), 
the  Computer  Select  system,  and  similar  sources. 

Interface  with  organizations  and  agencies  involved  in  the  systems  acquisition 
process  was  the  key  step  in  becoming  familiar  with  the  breadth  and  depth  of  M&S  use 
and  related  benefits,  problems,  and  successes.  By  understanding  the  underlying  policies 
and  procedures,  barriers,  and  metrics,  the  study  team  could  gather  the  information 
necessary  to  focus  on  actual  costs  and  benefits  of  M&S  in  DoD  acquisition.  While  the 
intuitive  and  probably  correct  answer  is  that  M&S  benefits  the  acquisition  process,  this 
study  sought  the  details  to  provide  a  more  quantitative  answer. 

During  several  visits  the  study  team  interfaced  with  subordinate  (labs,  agencies, 
divisions,  or  offices)  and  tenant  or  affiliated  (consortia,  contractors,  other)  groups 
collocated  with  or  near  the  main  organization.  On  most  visits,  the  study  team  was 
provided  ivith  significant  amoimts  of  material  tailored  to  the  questions  in  the  focus  paper, 
especially  pertaining  to  metrics,  new  and  evolving  M&S  tools,  challenges,  and 
costs/benefits.  The  team  developed  a  summary  of  the  visit  jfrom  notes  and  data  to  ensure 
historical  capture  of  key  insights  and  observations.  Copies  of  the  summaries  are  on  file. 

3.  Organizations  Visited 

Time  and  available  resources  limited  the  total  number  of  visits,  but  many  key 
organizations  and  individuals  were  visited  during  the  course  of  this  study.  Their 
experience  and  insights  were  essential  to  gaining  an  appreciation  of  the  benefits  and  the 
challenges  associated  with  M&S,  as  well  as  developing  a  method  to  best  quantify  the 
benefits  of  M&S  in  the  acquisition  process.  The  organizations  and  primary  POCs  are 
summarized  in  Table  II- 1 . 

While  specific  reports  of  these  visits  are  not  included  in  this  report,  the  data 
collected  during  the  visits  is  incorporated  and  referenced  throughout  Chapters  III  and  IV. 
We  are  grateful  for  the  courtesies  extended  and  the  information  shared  in  the  many  visits 
and  discussions. 


TABLE  II-l :  EXAMPLES  OF  KEY  VISITS 


SERVICE-LEVEL  POLICY  AND  MANAGEMENT: 


Deputy  Under  Secretary  of  the  Army  (Operations  Research) 

Mr.  Walt  Hollis 

Special  Assistant  for  M&S  to  Asst  Sec  of  the  Navy  (RDA) 

Mr.  Ben  Helme 

Air  Staff  (AF  T&E) 

Maj.  Bryan  Ishihara 

Office  of  Assistant  Secretary  of  the  Army  (RDA) 

Dr.  Herb  Fallin 

Defense  Advanced  Research  Projects  Agency  (DARPA) 

Dr.  Gary  Jones 

Army  Staff,  T&E  Management  Activity  (TEMA) 

Dr.  John  Foulkes 

Office  of  Assistant  Secretary  of  the  Navy  (RDA) 

Mr.  Mike  Roberts 

HQ,  Naval  Sea  Systems  Command 

Mr.  Dan  Billingsley 

_ R&D  AND  T&E  ACTIVITIES: 

Navy  M&S  War  Room  Project  Management  Ofc. _ 

USAF  Arnold  Engineering  Development  Center  (AEDC) 
USA  T&E  Command  (TECOM)  and  Aberdeen  Test  Center 
Patuxent  River  Naval  Air  Combat  Environment  T&E  Center 

US  Army  Redstone  Technical  Test  Center  (RTTC) _ 

USA  Tank  Automotive  RD&E  Center  (TARDEC) _ 

DoN/DoC  Center  of  Exc.  For  Best  Manufacturing  Practices 
Naval  Air  Warfare  Center  Weapons  Div  (NAWCWPNS) 

Air  Force  Flight  Test  Center  (AFFTC),  Edwards  AFB _ 

PROGRAM  OFFICES  AND  PEOs: 

F-22 

SC-21 _ 

AFATDS  (Advanced  Field  Artillery  Tactical  Data  System) 
Army  Tactical  Missiles  PEO  (Program  Executive  Office) 

COMANCHE _ 

JSF  (Joint  Strike  Fighter) 

TOMAHAWK 

NSSN  (New  Attack  Submarine  program) _ 

Javelin 

Longbow  Hellfire 

MLRS  (Multiple  Launch  Rocket  System) _ 

_ SAMPLE  INDUSTRY  VISITS: 

National  Automotive  Center  (NAC) _ _ 

Automotive  Research  Center  (ARC) _ 

Deneb  Robotics,  Inc 

Ford  Motor  Company 

Lockheed  Martin,  Palo  Alto,  CA 

Sikorsky  Helicopter 

General  Dynamics,  Electric  Boat  Div 

General  Motors,  Truck  &  Bus  Div 


B. 


Related  Studies 


Improvements  in  M&S  tools  to  support  acquisition  are  closely  interwoven 
with  initiatives  to  improve  the  overall  acquisition  process.  The  Packard  Commission 
highlighted  significant  process  flaws  in  their  1986  Report  to  the  Congress.  In  1993, 
then-Secretary  of  Defense  Aspin  and  others,  including  Dr.  John  M.  Deutch,  proposed 
acquisition  reforms  bolder  than  those  proposed  by  the  Packard  Commission.  Also  in 
1993,  now  Secretary  of  Defense  Dr.  Perry,  in  Acquisition  Reform:  A  Mandate  for 
Change,  described  the  problem  areas  in  the  acquisition  process  as  follows: 

DoD’s  acquisition  process  is  not  sufficiently  streamlined,  flexible,  agile, 
efficient,  timely  or  effective.  The  acquisition  community  has  not  been  sufficiently 
innovative,  has  not  used  technology  to  re-engineer  itself,  and  has  tended  to  use 
functional  stovepipes  rather  than  integrated  decision  teams. 

Numerous  recent  studies  have  been  conducted  which  deal  with  the  benefits  and 
limitations  in  DoD  use  of  M&S  in  support  of  the  acquisition  process.  Most  of  these 
studies  were  reviewed  for  their  applicability  to  this  effort  and  some  ideas  on  how  to 
measure  the  contribution  of  M&S  were  identified,  but  none  directly  addressed  the 
effectiveness  of  M&S  tools  and  processes  in  support  of  the  acquisition  process.  More 
comprehensive  summaries  can  be  found  in  Appendix  C. 

C.  Acquisition  Guidance 

1.  OSD  Guidance 

Before  examining  how  program  managers  are  employing  M&S,  it  is  appropriate 
to  review  the  guidance  given  by  OSD  to  the  Services,  and  in  turn  the  direction  the 
Services  give  to  the  acquisition  managers. 

The  acquisition  environment  has  many  new  challenges  including  downsizing, 
reduced  funding,  and  the  need  to  reduce  the  time  and  resources  needed  for  integrating 
advanced  technologies  into  new  systems.  The  goal  is  to  procure  state-of-the-art 
technology  and  products  ‘better,  faster,  and  cheaper’  while  also  helping  to  transition  DoD 
and  its  contractors  to  the  new  methods  and  tools  that  enable  advanced  acquisition. 

Secretary  of  Defense  Perry  recently  spoke  at  an  Operational  Test  Agency  (OTA) 
Commander’s  Conference  and  discussed  five  key  themes:  better  use  of  M&S;  focus 
testing  where  M&S  can’t  be  applied;  combine  tests  (developmental  and  operational,  or 
operational  testing  for  two  or  more  systems)  when  possible;  combine  testing  with  training 
or  operations;  and  early  involvement  of  testers. 

In  his  keynote  address  to  the  Defense  Modeling  and  Simulation  Office  Fifth 
Annual  Industry  Briefing  on  Modeling  and  Simulation  in  May  1996,  the  Under  Secretary 


of  Defense  (USD),  Acquisition  and  Technology  (A&T),  Dr.  Paul  G.  Kaminski,  discussed 
the  critical  and  increasing  role  of  modeling  and  simulation  in  acquisition.  He  stated  that: 

The  bottom  line  is  that  integrated  product  and  process  development, 
backed  up  by  a  strong  commitment  to  computer  based  modeling  and  simulation 
tools,  provides  a  dominant  competitive  edge  in  the  commercial  marketplace  and  a 
clear  warfighting  edge  on  the  battlefield.  It  provides  a  path  for  getting  to  market 
first  and  at  a  lower  cost. 

A  recent  interview  with  Philip  Coyle,  Director  of  Operational  Test  and  Evaluation 
in  OSD,  in  Program  Manager  magazine  highlights  some  key  areas  of  M&S  emphasis. 

The  key  is  to  make  testing  more  efficient.  This  can  happen  by  involving  testers  early  in 
the  acquisition  process,  starting  before  the  request  for  proposal  (RFP)  and  during 
development  of  the  operational  requirements  document  (ORD).  It  also  requires  involving 
the  OTAs  and  the  Joint  Interoperability  Test  Center  (JITC)  up  front.  He  also  identified 
the  need  for  better  use  of  M&S  including  tools  that  are  more  predictive  (high  probability 
of  giving  the  right  answer  as  a  result  of  being  based  on  real  science,  or  ‘physics-based’). 

DoD  Directive  5000.1  encourages  the  use  of  M&S  and  states: 

Models  and  simulations  shall  be  used  to  reduce  the  time,  resources,  and 
risks  of  the  acquisition  process  and  to  increase  the  quality  of  the  systems  being 
acquired.  Representations  of  proposed  systems  (virtual  prototypes)  shall  be 
embedded  in  realistic,  synthetic  environments  to  support  the  various  phases  of  the 
acquisition  process,  firom  requirements  determination  and  initial  concept 
exploration  to  the  manufacturing  and  testing  of  new  systems,  and  related  training. 

In  addition,  DoD  5000.2-R  states: 

Accredited  modeling  and  simulation  shall  be  applied,  as  appropriate, 
throughout  the  system  life-cycle  in  support  of  the  various  acquisition  activities: 
requirements  definition;  program  management;  design  and  engineering;  efficient 
test  planning;  result  prediction;  and  to  supplement  actual  test  and  evaluation; 
manufacturing;  and  logistics  support.  PMs  shall  integrate  the  use  of  modeling  and 
simulation  within  program  planning  activities,  plan  for  life-cycle  application, 
support,  and  reuse  models  and  simulations,  and  integrate  modeling  and  simulation 
across  the  functional  disciplines. 

The  DoD  M&S  Master  Plan  (DoD  5000. 59-P)  states  the  vision  as  follows: 

Defense  modeling  and  simulation  will  provide  readily  available, 
operationally  valid  environments  for  use  by  the  DoD  Components: 

-To  train  jointly,  develop  doctrine  and  tactics,  formulate  operational  plans, 
and  assess  warfighting  situations. 


-To  support  technology  assessment,  system  upgrade,  prototype  and  full- 
scale  development,  and  force  structuring. 

-Furthermore,  common  use  of  these  environments  will  promote  a  closer 
interaction  between  the  operations  and  acquisition  communities  in  carrying  out 
their  respective  responsibilities.  To  allow  maximum  utility  and  flexibility,  these 
modeling  and  simulation  environments  will  be  constructed  from  affordable, 
reusable  components  interoperating  through  an  open  systems  architecture. 

2.  Service  Guidance 

Guidance  for  the  use  of  M&S  in  weapon  systems  acquisition  in  the  Services 
varies.  The  Navy  employs  different  organizations  to  address  oversight,  policy,  technical 
support,  and  use  of  M&S,  because  of  the  breadth  of  functional  disciplines  which  Navy 
M&S  supports.  US  Marine  Corps  M&S  efforts  are  integrated  into  portions  of  the  Navy’s 
M&S  structure  and  there  is  also  a  steering  group  and  management  office  for  USMC- 
specific  functions  and  systems. 

The  Navy  is  establishing  a  PEG  for  Synthetic  Operations,  but  the  functions  and 
responsibilities  of  this  office  have  yet  to  be  approved.  The  Navy  also  has  a  War  Room 
for  M&S  for  better  integration  of  M&S  functions  and  systems.  Designed  to  work  on 
M&S  planning  and  implementation  across  the  Navy,  from  initial  requirements  to  system 
logistical  support,  the  War  Room  also  works  on  short  term  projects  such  as  the  Navy 
M&S  Master  Plan  and  the  Navy’s  VV&A  Instruction. 

The  Army  Model  and  Simulation  Office  (AMSO)  was  charged  with  producing  an 
investment  plan  that  will  focus  on  efficient  M&S  efforts.  RDA  is  one  of  three  domains 
established  to  support  this  office.  This  initiative  was  begun  in  1995  and  management  is 
being  implemented. 

Office  of  the  Assistant  Secretary  of  the  Army  (OASA)  for  RDA  policy 
memorandum,  “Simulation  Support  to  Army  Acquisition,”  mandates  use  of  a  simulation 
support  plan  (SSP)  for  each  Army  ACAT I  and  II  program  prior  to  each  milestone 
review.  The  long-term  goal  is  to  eliminate  use  of  a  separate  SSP  and  integrate  the  M&S 
related  information  into  the  Acquisition  Strategy  Report.  This  will  be  another  step  in 
better  integration  of  M&S  into  the  overall  acquisition  process. 

The  Air  Force  has  a  long  history  of  M&S  applications  and  a  growing  need  to 
improve  on  the  use  of  M&S  for  decision-making.  Significant  organizational  changes 
occurred  in  late  1984.  The  Directorate  of  Modeling,  Simulation  and  Analysis  (MS&A) 
(HQ,  USAF/XOM)  was  designated  the  single  point  of  contact  for  M&S.  Within  this 
organization  several  divisions  provide  support  in  the  areas  of  evaluation,  technical 
matters,  warfighting,  studies  and  analysis,  operational  requirements,  and  integrated 
product  planning. 


Most  recently,  the  Air  Force  published  a  Functional  Area  Plan  for  Modeling  and 
Simulation.  Together  with  the  New  Vector  initiative  to  guide  Air  Force  M&S  and  the  Air 
Force  M&S  Master  Plan,  published  earlier  this  year,  the  Functional  Area  Plan  provides 
the  framework  to  integrate  M&S  into  a  single  coordinated  program. 

D.  The  Traditional  Acquisition  Process 

This  section  briefly  summarizes  the  phases  and  milestones  of  the  traditional 
acquisition  process  in  order  to  create  a  baseline  for  comparison  with  the  evolving  process. 
The  traditional  process  consists  of  distinct  phases,  starting  with  requirements  generation 
or  determination  of  mission  need,  and  ending  with  the  operations  and  support  and 
eventually  the  retirement  of  the  system.  The  phases  are  separated  by  milestone  decision 
points  where  key  decisions  are  made.  These  Milestone  Reviews  were  often  characterized 
as  walls  between  phases  which  fostered  a  stovepiped  approach  that  was  evident 
throughout  the  process. 

Most  of  the  risk  assessment  was  conducted  early  in  the  program,  requiring  use  of 
imperfect  assumptions  and  information  (especially  with  regard  to  cost  and  technology). 

A  result  was  imperfect  expectations,  and  little  flexibility  in  the  system  after  the  early 
stages  of  the  program. 

Members  of  the  acquisition  community  who  worked  within  this  traditional 
process  operated  within  their  own  sphere  of  influence  with  very  little  coordination  before 
handing  off  the  product  to  the  next  phase.  This  often  fostered  a  situation  in  which 
different  groups  or  teams  had  adversarial  relationships.  The  Government,  especially  the 
user  community,  had  very  high  expectations  for  the  technology  and  the  ability  of  the 
contractor  to  achieve  success,  sometimes  unrealistically  so.  The  test  community  played 
an  oversight  role  ensuring  that  the  evolving  system  met  the  user  requirements  to  a 
suitable  degree.  On  the  other  hand,  the  contractor  and  program  manager  sought  to  move 
the  program  along  to  the  fielding  or  initial  operational  capability  (IOC)  stage.  There  was 
infrequent  interaction  between  the  various  participants.  User  discovery  of  performance  or 
capability  shortfalls  after  fielding  often  led  to  expensive  Engineering  Change  Proposals 
(ECPs)  and  modifications. 

The  traditional  system  resulted  in  a  good  examination  of  the  final  product,  but 
failed  to  maximize  efficiency,  especially  in  light  of  the  timing  of  requirements 
specification.  While  some  M&S  tools  have  long  been  employed  in  the  acquisition 
process,  they  too  were  used  in  a  stovepipe  fashion  with  little  or  no  use  of  common  data 
across  fimctional  areas  or  across  programs.  Tools  that  are  currently  available  to  examine 
the  system  in  a  virtual  manner  were  not  available,  thus  a  physical  prototype  was 
necessary  to  examine  fit  and  fimction  issues  in  detail  before  readiness  for  production. 

Before  the  advent  of  integrated  teams  in  the  weapon  systems  development 
process,  there  was  little  effort  made  to  combine  testing  to  satisfy  the  different 


requirements,  or  to  collect  extra  data  in  early  efforts  so  they  could  be  used  as  part  of  the 
database  later  in  the  acquisition  cycle.  Dwindling  resources  and  integrated  teams  have 
made  program  managers  more  aware  of  the  value  of  integrating  workers  from  different 
phases  and  using  simulations  and  models  to  accomplish  multiple  missions. 

In  spite  of  being  bound  by  processes  and  Milestone  Reviews,  the  community  is 
evolving  the  program  offices  and  their  support  into  an  information  team  that  shares  data 
and  results  across  the  phases  of  the  acquisition  process.  The  positive  results  of 
simulation  efforts  in  systems  engineering  have  become  evident  in  what  we  refer  to  as 
Simulation  Based  Acquisition.  The  developments  and  trends  in  each  phase  of  the 
acquisition  process  that  are  affected  by  the  immersion  in  M&S  use  are  discussed  in  the 
next  chapter. 


III.  SIMULATION  BASED  ACQUISITION 


A.  Introduction  to  Simulation  Based  Acquisition 

M&S  tools  have  been  used  to  support  the  systems  acquisition  process,  both  in  DoD  and 
elsewhere,  for  a  long  time.  As  dramatic  advances  in  the  supporting  technologies  made  those 
tools  more  powerful  and  less  expensive,  and  as  declining  resources  and  changing  priorities  made 
it  essential  to  find  better  ways  to  develop  and  field  new  systems,  the  use  of  these  tools  and  of 
improved  processes  that  exploit  their  contribution  has  expanded  rapidly.  It  is  not  the  purpose  of 
this  report  to  justify  why  the  emerging  M&S  tools  and  processes  should  be  used,  but  rather  to 
cite  documented  contributions  to  the  total  acquisition  process. 

This  use  of  M&S  tools  has  increased  in  an  evolutionary  manner  so  that  many  would 
observe  from  their  experience  that  ‘nothing  has  changed,  we  have  always  used  M&S  tools.’  In 
fact,  much  has  changed  and  the  rate  of  that  change  is  increasing  rapidly.  This  increase  has  not 
been  imposed  by  fiat;  it  is  not  the  result  of  new  guidance  or  direction  from  top  management. 
Rather  it  is  the  result  of  piecemeal  adoption  of  powerful  new  emerging  M&S  tools  to  support 
existing  processes  and  to  satisfy  emerging  requirements. 

As  a  result,  these  tools  have  been  employed  differently  by  acquisition  programs  at  various 
stages  of  development  in  a  seemingly  ad  hoc  manner.  Some  programs  are  little  changed  from 
their  processes  of  a  decade  ago  while  others  are  being  radically  transformed  by  changing 
processes  that  depend  upon  these  tools.  Most  programs  fall  in-between,  using  M&S  tools  at 
some  stages  of  their  program’s  development  with  modification  of  existing  processes.  Program 
managers  are  utilizing  M&S  tools  in  areas  where  they  see  the  greatest  benefit. 

Although  it  is  not  clear  where  this  will  lead  the  acquisition  community  in  the  next  decade, 
it  is  clear  that  a  revolution  is  underway  and  that  the  end  result  will  be  a  new  way  of  doing 
business.  We  will  call  this  new  approach  to  acquisition,  “Simulation  Based  Acquisition,”  in 
order  to  differentiate  it  from  the  traditional  approaches  (Chapter  II)  and  to  emphasize  its  reliance 
on  the  tools  and  processes  made  possible  by  advances  in  simulation  technology.  It  is  not 
presented  here  as  a  specific,  definable  concept,  but  rather  as  a  term  to  characterize  the  general 
approach  of  significantly  increased  use  of  M&S  tools  and  the  new  processes  which  they  enable 
in  a  new,  more  integrated  approach  to  program  development. 

One  area  that  can  be  impacted  in  a  positive  manner  by  the  new  acquisition  approach  is 
life  cycle  cost.  It  is  well  established  that  a  large  percentage  of  total  system  cost  is  determined  by 
initial  design  decisions.  Early  system  research  and  development  (R&D)  is  usually  only  about 
10%  of  total  system  cost  but  establishes  the  level  of  future  production  and  operations  cost. 
Relatively  small  investments,  one  or  two  percent  of  the  life  cycle  cost  of  the  system,  could  result 
in  significant  savings  (greater  than  25%)  over  the  system’s  life.  The  traditional  acquisition 
approach  lacked  the  tools  to  conduct  “what  if’  analyses  to  determine  the  impact  of  design 
decisions  on  future  cost.  These  types  of  tools  are  now  available. 


The  charter  of  this  study  was  to  document  metrics,  benefits,  and  problems  associated  with 
the  use  of  these  M&S  tools  in  the  acquisition  process.  To  keep  this  effort  focused,  system 
requirements  analysis  and  training  and  training  device  development  were  not  included  in  the 
scope  of  this  tasking.  The  programs  investigated  in  this  effort  were  at  different  stages  of 
development,  with  none  having  used  these  tools  yet  in  an  integrated  fashion  throughout  their 
development  cycle.  This  study  is  consequently  a  series  of  independent  stories,  told  in  a  familiar 
context.  That  framework  is  the  acquisition  process:  concept  development,  engineering  design 
and  analysis,  test  and  evaluation,  manufacturing,  and  deployment  and  support. 

The  Simulation  Based  Acquisition  (SBA)  approach  can  generally  be  characterized  as 
more  flexible  and  integrated  than  the  previous  approaches  which  are  often  thought  of  as 
stovepipes.  In  a  conceptual  way.  Figure  III-l  shows  the  functions  of  the  acquisition  process 
without  the  fixed  boundaries  and  with  emphasis  on  their  concurrent,  integrated  development. 
There  is  increasing  interaction  among  all  of  the  functions,  a  genuine  concurrent  approach,  with 
data  and  expertise  being  shared  and  interacting  throughout  the  entire  process. 
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Figure  III-I .  Simulation  Based  Acquisition 


The  savings  in  simulation  based  acquisition  are  not  always  easy  to  quantify.  As  more 
programs  document  their  success  and  experience  using  M&S  in  the  acquisition  process,  there 
emerge  four  bins  into  which  these  measures  of  effectiveness  can  be  categorized.  These  four 
areas  of  realized  benefits  are  cost,  schedule,  productivity,  and  quality/performance. 

It  is  worth  noting  that  many  reported  cost  savings  are  often  illusionary.  They  are 
typically  more  correctly  classified  as  “cost  avoidance.”  As  such,  they  are  measures  of  significant 
additional  work  or  results  that  were  obtained  using  the  M&S  tools  which  would  have  cost  the 
reported  “savings”  if  they  had  been  obtained  by  more  traditional  methods.  This  study  has  been 
careful  to  differentiate  between  cost  savings  and  cost  avoidance  and  to  clearly  identify  the  latter 
category  when  reported.  Even  if  cost  avoidance  “savings”  should  not  be  properly  measured  in 
dollars,  they  almost  always  are  an  indication  of  improved  system  quality  or  performance  or 
lowered  program  risk  (e.g.,  smarter  testing).  In  some  cases,  the  program  itself  would  not  be 
economically  feasible  without  these  improved  methods  which  provide  adequate  risk  reduction 
for  an  affordable  cost. 

The  approach  of  this  chapter  is,  for  each  function  of  the  development  (systems 
engineering)  process,  to: 

•  describe  how  M&S  tools  are  changing  the  acquisition  process, 

•  identify  metrics  which  validate  the  effectiveness  of  these  tools  in  this  process,  and 

•  relate  specific  quantified  examples  of  benefits  of  M&S  tools  in  this  process. 

While  the  following  benefits  are  described  within  a  functional  area  because  of  their 
similarity,  and  to  emphasize  the  value  of  M&S  tools  and  processes  within  that  function,  it  is 
recognized  that  product  cycle  time  is  a  major  (top  level)  metric  of  great  significance.  For 
example,  the  US  automobile  industry  (Big  Three)  report  that  with  use  of  M&S  tools  and 
processes,  they  have  reduced  the  cycle  time  (from  concept  approval  to  production)  from  five 
years  down  to  three  years,  and  expect  significant  further  reductions.  Electric  Boat  has  reported 
reducing  the  cycle  time  for  submarine  development  from  14  years  down  to  7  years. 

The  next  chapter  will  relate  technical,  cultural,  and  managerial  challenges  which  prevent 
programs  from  realizing  the  full  benefit  from  these  M&S  tools. 

B.  Concept  Development  Process 

Concept  development  is  the  analysis  of  operational  requirements  and  their  transformation 
into  functional  requirements  for  a  new  system.  Alternative  approaches  to  satisfying  the 
requirements  are  evaluated  to  select  the  best  general  approaches  to  a  system  solution. 

1 .  Tools  in  the  Evolving  Concept  Development  Process. 

Because  this  stage  of  development  is,  by  nature,  almost  totally  conceptual,  it  has  always 
relied  upon  graphical  and  data  representations  (models)  of  proposed  alternatives.  Computer  tools 


to  support  this  process  have  benefited  from  the  recent  growth  in  power  and  reduction  in  price  of 
hardware  and  software  which  allows  the  cost  effective  analysis  of  far  more  alternatives  than  was 
possible  previously.  More  significantly,  M&S  tools  have  benefited  from  efforts  to  integrate 
related  tools  which  enable  more  extensive  analysis  of  alternatives  in  a  more  realistie  operational 
environment.  Though  the  operational  environment  is  virtual,  recent  progress  in  developing  and 
making  available  appropriate  realistic  and  validated  virtual  environments  has  been  significant 
and  provides  the  necessary  credibility  to  emerging  processes  that  use  these  tools. 

The  ability  to  develop  digital  master  product  models  with  Computer-Aided  Design/ 
Computer-Aided  Manufacturing/Computer- Aided  Engineering  (CAD/CAM/CAE)  software 
teehnology  has  made  it  possible  to  more  fully  understand  products  from  a  manufacturing 
standpoint  during  the  early  design  stage.  The  depth  and  richness  of  the  product  information 
contained  in  a  digital  master  model  makes  it  easier  to  more  fully  communicate  detailed 
information  to  everyone  involved  in  the  product  development  process. 

Digital  master  models  help  to  develop  and  evaluate  multiple  design  concepts  so  that  the 
material  solution  most  efficiently  meets  user  needs.  Quality  becomes  part  of  the  design  process 
itself  and  can  be  built-in  instead  of  added-on.  Digital  master  models  provide  details  about  the 
product’s  shape,  behavior,  and  cost  before  the  fabrication  of  costly  physical  prototypes,  and  help 
minimize  scrap,  reduce  downtime,  and  eliminate  wasted  or  redundant  operations.  This  approach 
allows  teams  to  work  concurrently  by  providing  common  ground  for  interrelated  product 
development  tasks.  Instead  of  individuals  creating  one  piece  of  information  at  a  time,  the  digital 
master  model  enables  various  disciplines  to  work  together  much  earlier  in  the  product 
development  process. 

One  intangible,  but  important  benefit  of  the  new  M&S  tools  is  the  ability  to  quickly  and 
effectively  communicate  information  on  how  a  new  concept  will  look  and  function  to  senior 
decision  makers. 

2.  Metrics. 

-  money/effort  saved  in  programs  that  are  canceled/terminated  early 

-  rapid  and  early  proof  of  concept  prior  to  more  detailed  stages  of  design 

-  better  input  from  users  on  the  impact  of  concept  design  decisions 

3.  Benefits  of  M&S. 

a.  Several  joint  and  Air  Force  warhead/munitions  programs  were  terminated  early  in 
concept  development,  before  expending  resources  wastefully,  due  to  improved  analysis  of 
system  lethality  using  M&S  tools: 

•  Wide  Area  Anti- Armor  Munitions  (WAAM)  -  $30M  program, 

•  Hypervelocity  Missile  -  $  1 OM  program, 

•  Kinetic  Energy  Penetrator  (KEP)  -  $50M  program. 


•  JP-233  Runway  Attack  Munition  -  $54M  program,  and 

•  Boosted  Kinetic  Energy  Penetrator  (BKEP)  -  $130M  program. 

b.  Vehicle  concepts  have  been  terminated  or  not  further  explored  due  to  mobility, 
firepower,  or  vulnerability  problems  discovered  early  using  TARDEC’s  Tracked  Vehicle  Work 
Station  (TVWS),  which  creates  and  evaluates  new  concepts  in  a  virtual  environment. 

c.  Several  Navy  programs  (NSSN,  LPD-17)  have  explored  initial  concepts  for 
overall  viability  and  crew/functional  acceptability  (sizing,  location)  prior  to  further  design  effort. 
Examples  are  crane  selection  and  helicopter  control  station  window  design  for  the  LPD-17,  along 
with  assessments  of  cargo  locations  and  on-board  vehicle  movement.  For  NSSN,  maintenance 
accessibility  for  personnel  and  for  machinery  removal  was  evaluated  early  in  the  design. 

d.  The  Department  of  Energy  (DoE)  at  Oak  Ridge,  TN,  reported  analyzing  many 
‘bright  ideas’  with  M&S  tools  since  they  were  too  expensive  to  physically  check  out. 

Ultimately,  all  the  concepts  were  rejected  as  flawed  based  upon  these  analyses. 

C.  Engineering  Design  and  Analysis  Process 

The  design  process  is  well  understood  from  its  long  tradition  of  engineering  education 
and  experience.  The  graphical  and  computational  tools  that  support  it  have  evolved  slowly,  but 
the  computer  revolution  has  also  enabled  significant  changes  in  recent  years.  Slide  rules  have 
essentially  disappeared  for  faster  and  more  reliable  computational  tools,  but  engineering 
drawings  and  representations  of  components,  subsystems,  and  systems  as  a  means  to 
communicate  system  design  are  still  widely  used. 

1 .  Tools  in  the  Evolving  Engineering  Design  and  Analysis  Process. 

This  past  decade  has  witnessed  the  emergence  of  computer  tools  to  assist  this  process, 
primarily  in  the  domain  of  CAD/CAM/CAE  tools.  These  tools  automated  some  of  the  manual 
tasks  of  developing  graphical  and  data  representations  of  systems,  but  they  remained  at  the 
individual  engineer  or  small  functional  design  team  level  of  use. 

Driven  by  the  commercial  marketplace,  these  CAD/CAM/CAE  tools  have  improved 
greatly,  but  it  is  the  integration  of  these  tools  and  the  changing  of  processes  which  utilize  these 
tools  that  is  beginning  to  produce  significant  results.  When  the  entire  design  team,  to  include 
manufacturing,  support,  and  test  activities,  as  well  as  subcontractors,  are  linked  in  a  meaningful 
fashion  to  this  common  database,  a  genuine  concurrent  approach  to  development  is  possible. 

The  various  members  of  this  design  team  no  longer  need  to  be  collocated.  Technology  now 
allows  each  to  work  from  their  own  locations  in  an  effective,  distributed  manner.  Far  more 
design  alternatives  can  now  be  developed  and  analyzed  for  performance  and  cost  impacts  before 
finalizing  design  decisions. 


In  addition  to  an  integrated  team  approach  to  system  development,  as  confidence  in  the 
validity  and  accuracy  of  these  tools  has  increased,  the  requirements  for  elaborate  and  expensive 
physical  prototypes  of  the  new  system  have  relaxed.  Very  expensive  physical  prototypes  are  no 
longer  the  only  way  to  ensure  that  integration  and  fit  requirements  have  been  satisfied.  Virtual 
prototypes  are  proving  to  be  acceptable  representations  for  evaluation  of  many  system  integration 
issues. 


2.  Metrics. 

-  number  of  people  required  to  accomplish  design  tasks 

-  time  required  to  accomplish  design  tasks 

-  number  of  physical  mockups  or  prototypes  that  are  required 

-  number  of  parts  or  complexity  of  the  final  design  (i.e.,  simplicity) 

-  reduction  in  first  article  assembly  effort 

3 .  Benefits  of  M&S . 

a.  In  the  DARPA  Initiative  for  Concurrent  Engineering  (DICE)  program,  TRW 
Corporation  redesigned  a  radar  warning  system  using  two  different  approaches;  traditional  and 
concurrent  design  with  integrated  design  automation.  The  traditional  approach  required  96  man 
months  while  the  advanced  approach  required  only  46  man  months. 

b.  TARDEC  prepared  an  engine  replacement  analysis  for  the  Bradley  Fighting 
Vehicle  (BFV)  system  using  half  a  man  month  with  M&S  tools  compared  to  the  traditional 
approach  which  required  4-6  man  months  to  accomplish. 

c.  TARDEC  designed  a  new  low  silhouette  tank  prototype  using  14  engineers  in  16 
months.  Traditional  approaches  have  required  about  55  engineers  and  3  years. 

d.  General  Electric  used  a  new  parametric  modeling  approach  to  design  a  new 
engine  fan  blade,  thereby  reducing  the  process  time  from  up  to  four  weeks  to  a  few  hours. 

e.  New  integration  techniques  employed  by  Martin  Marietta  (now  Lockheed  Martin 
Corporation)  using  M&S  tools  and  processes  resulted  in: 

•  reduction  of  engineering  time  to  construct  mock-ups  from  2,100  hours  to  900  hours  , 

•  cutting  the  number  of  tool  designers  required  from  10  to  4, 

•  reduction  of  physical  electronic  board  development  iterations  from  2.5  to  1.5,  and 

•  reduction  of  the  number  of  changes  per  final  drawing  from  4  to  2.2  for  a  savings  of 
about  $108M  per  year. 

f  Lockheed  Martin  used  solid  modeling  and  computer  simulated  assembly  for 
aircraft  parts  design  which  allowed: 

•  elimination  of  some  physical  mockups  at  $30M  each. 


•  cost  reduction  in  design/verification  steps  of  30-50%,  and  reduction  in  first  article 
(and  subsequent)  assembly  costs  of  20-25%. 

g.  The  AS400  computer  development  program  by  IBM  used  extensive  simulation 
and  cross-functional  teams  to  simplify  design  and  reduce  the  part  count  to  4000  as  opposed  to 
10,000  for  the  earlier  S/38  computer.  They  used  a  simulator  to  test  software  design  alternatives 
until  they  converged  upon  a  virtually  defect-free  design,  and  did  this  in  about  two  years  -  less 
than  half  the  time  required  for  its  predecessor  machine.  About  10  months  of  the  two-year  design 
cycle  reduction  was  due  to  the  use  of  design  simulation. 

h.  Through  advanced  integration  techniques.  Motorola  achieved  the  following  gains: 

•  a  50%  reduction  in  design  cycle  time  for  end  products  using  Application  Specific 
Integrated  Circuits, 

•  five-week  to  one-week  reduction  in  printed  circuit  board  supply  and  assembly  cycle, 

•  cellular  communications  product  cycle  reduced  50%. 

i.  Sikorsky  Aircraft  employed  38  draftsmen  for  approximately  six  months  to 
produce  the  working  drawings  of  the  CH-53E  Super  Stallion  aircraft’s  outside  contours.  The 
same  task  on  the  Comanche  program  only  required  one  month  for  one  engineer  using  M&S 
tools.  In  addition,  precisely  designed  parts  from  different  contractors  assembled  properly  on  the 
first  attempt  and  required  a  minimum  amount  of  rework. 

j  ■  The  Comanche  PM  Office  mandated  early  use  of  mission  and  engineering 

simulators  to  examine  operational  characteristics.  The  Comanche  Simulator  and  Surrogate 
Aircraft  Fly-Off  for  source  selection  cost  about  $20  million,  versus  about  $500  million  for  the 
prototype  aircraft  fly-off  for  the  UH-60  Blackhawk 

k.  Seakeeping  analysis  in  support  of  ship  design  by  Naval  Sea  Systems  Command 

(NAVSEA)  in  1985  took  about  27  days.  In  1994,  the  same  analysis  was  conducted  in  3.5  days 
using  better  M&S  tools  operating  interactively. 

l.  Radar  cross  section  (RCS)  analysis  and  treatment  for  ship  design  by  NAVSEA 
took  about  57  days  in  1990.  Using  new  M&S  tools  and  processes,  the  Navy  now  can  conduct  the 
same  analysis  in  16.75  days. 

m.  Newport  News  Shipbuilding  reported  improved  M&S  tools  and  processes  reduced 
engineering  design  time  by  40%  and  tooling  time  by  23-27%. 

n.  Boeing  designed  a  new  strut  for  the  767  using  an  all-digital  data  process  in  1 7% 
less  time,  saving  30,000  man-hours,  than  previous  approaches.  They  also  had  a  65%  decrease  in 
design  changes. 

o.  LPD-1 7  saved  $6  million  in  design  costs  by  using  new  M&S  tools.  It  also  saved 
100  tons  in  topside  weight  which  is  expected  to  greatly  improve  performance. 


p.  Boeing’s  777  program  exceeded  its  goal  of  reducing  change,  error,  and  rework  by 
50%.  Parts  and  systems  fit  together  better  than  anticipated  and  at  the  highest  level  of  quality. 

On  the  first  777  aircraft  assembled,  the  final  body  joining  tolerance  was  just  0.023  inch  away 
from  a  perfect  alignment  compared  to  the  typical  half  inch  experienced  in  previous  programs. 

D.  Test  and  Evaluation  Process 

Testing  new  systems  has  long  involved  using  the  production  hardware  in  a  live 
environment.  Testing  of  defense  systems  for  their  ability  to  meet  operational  requirements  has 
emphasized  the  use  of  typical  operators  in  an  operationally  realistic  scenario  and  environment. 
The  increasing  use  of  simulators  to  focus  the  scope  and  reduce  the  cost  of  testing  actual  hardware 
and  software  demonstrate  early  movement  toward  the  incorporation  of  M&S  tools  and  processes. 

1 .  Tools  in  the  Evolving  T&E  Process. 

The  senior  levels  of  the  Defense  Department  are  now  consistent  in  their  support  of 
increased  use  of  M&S  tools  in  the  T&E  process  because  of  the  promise  inherent  in  those  tools  to 
help  meet  the  challenges  of  reduced  resources  mentioned  above  as  well  as  focusing  the  scope  and 
improving  the  quality  of  the  test  process.  In  recent  years  there  has  been  continuing  dialogue 
about  how  to  best  incorporate  those  tools,  with  the  underlying  issue  of  the  credibility  of  the  tools 
being  a  major  constraint.  At  a  minimum,  the  early  involvement  of  the  T&E  engineer  in  the 
concept  development  stage,  to  ensure  that  functional  requirements  are  formulated  in  a  testable 
manner,  is  facilitated  and  aided  by  M&S  tools.  Similar  tools  are  now  widely  used  to  plan, 
rehearse,  extend,  and  evaluate  live  T&E  activities.  To  a  significant  extent,  assessment  of  a 
system  is  now  possible  using  M&S  long  before  a  physical  prototype  is  actually  constructed. 
Physics-based  dynamic  models  using  CAD  descriptions  of  the  system  in  a  test-validated 
environment  now  provide  critical  feedback  to  system  designers,  as  well  as  users,  as  the  design 
matures. 

For  example,  the  first  flight  of  a  new  aircraft  is  preceded  by  years  and  hundreds  of 
millions  of  dollars  of  mathematical  modeling,  flight  simulations,  hardware-in-the-loop  (HWIL), 
software-in-the-loop  (SWIL),  and  other  simulations. 

M&S  is  being  integrated  with  testing  as  part  of  a  strategy  to  provide  information 
regarding  risk  and  risk  mitigation,  to  provide  empirical  data  to  validate  M&S,  to  permit  an 
assessment  of  the  attainment  of  technical  performance  specifications  and  system  maturity,  and  to 
support  determination  of  whether  systems  are  operationally  effective,  suitable,  and  survivable  for 
intended  use.  In  operational  testing,  M&S  is  used  to  complement  the  testing  of  systems  in  an 
operational  environment  by  extending  the  scenario  or  the  environment  to  make  it  possible  to 
assess  additional  situations. 

The  Air  Force  Flight  Test  Center  (AFFTC)  at  Edwards  AFB  reports  a  situation  common 
to  other  Research,  Development,  Test  and  Evaluation  (RDT&E)  Centers  and  PM  Offices.  The 
“savings”  resulting  from  use  of  improved  M&S  tools  and  processes  is  not  realized  in  dollars,  or 


in  less  test  flights  or  test  time,  but  rather  used  to  address  additional  issues  that  otherwise  would 
not  be  tested  because  of  limited  flying  hours  and  resources  available. 

One  of  the  primary  benefits,  especially  for  aircraft  test  centers,  is  avoiding  the  “cost  of 
failures.”  This  cost  avoidance  is  extremely  important  to  the  development  programs,  and  includes 
the  incalculable  cost  of  risking  the  loss  of  life  of  test  personnel.  The  following  hypothetical 
example  of  a  missile  program  will  illustrate  some  aspects  of  the  “cost  of  failure.” 

Assume  that:  a  flight  failure  causes  a  two  month  program  delay;  the  cost  associated  with 
this  delay  is  the  monthly  spend  rate  times  two;  the  monthly  spend  rate  is  $5  million;  40  live 
missile  flights  are  required  for  the  test;  there  is  a  90%  success  rate  of  missile  flights.  Then  a  two 
month  slip  will  cost  the  program  $10  million  and  four  failures  (40*  [1-0.9])  are  expected  for  a 
total  program  cost  of  $40  million.  If  the  M&S  tools  are  able  to  prevent  half  of  the  failures,  a  cost 
savings  of  $20  million  is  realized. 

2.  Metrics. 

-  number  of  areas  assessed  that  are  difficult/impossible  to  test  physically  due  to 
limitations  in  cost,  time,  or  manpower,  or  due  to  risk  to  humans,  equipment,  or  the 
environment 

-  quality  and  quantity  of  test  data  gathered 

-  amount  of  testing  concurrent  with  development 

-  stimulation  realism  (creating  a  realistic  number  of  effects  with  fidelity) 

-  cost  avoidance  (for  test  article  as  well  as  test  resources) 

-  number  of  programs  retained  (i.e.,  avoid  program  termination) 

-  number  of  safety  problems  resolved  before  human  testing 

3.  Benefits  of  M&S. 

a.  The  Tactical  Missile  Signature  (TMS)  center  and  database  at  Arnold  Engineering 
Development  Center  (AEDC)  reports  that  obtaining  the  measurements  for  various  missile 
characteristics  can  cost  up  to  $700,000  per  missile  to  set  up,  fire,  and  collect  and  reduce  the  data. 
Equivalent  data  can  be  obtained  using  M&S  tools  for  less  than  $10,000. 

b.  The  F-22  program  used  AEDC’s  integrated  T&E  (IT&E)  concept  to  address  the 
need  to  safely  separate  fuel  tanks  and  weapons  from  the  aircraft  over  a  range  of  flight  conditions. 
Modeling  of  the  pylons  and  the  maneuvering  aircraft  were  combined  and  compared  to  wind 
tunnel  model  scale  data.  The  degree  of  correlation  validated  the  approach  and  allowed  ground 
simulation  results  equivalent  to  flight  conditions.  The  IT&E  approach  met  the  goals  of  reduced 
cost  and  increased  information  to  reduce  program  risk.  In  addition  to  providing  the  information, 
this  approach  saved  the  F-22  program  a  verified  $8  million. 

c.  AEDC  used  combined  computational  fluid  dynamics  (CFD)  and  engineering 
methods  to  predict  the  separation  of  the  Pegasus  XL  rocket  booster  from  an  L-101 1  Tristar 
aircraft  without  having  to  conduct  wind  tunnel  testing  that  would  have  cost  about  $750,000. 


d.  Testing  of  stockpiled  munitions,  for  systems  such  as  HELLFIRE  and  Javelin,  is 
required  to  ensure  their  reliability  after  storage.  M&S  techniques  coupled  with  non-destructive 
electronic  component  testing  has  dramatically  reduced  the  number  of  live  missile  firings  required 
to  certify  reliability  of  a  stockpiled  lot. 

e.  Advanced  Medium  Range  Air-to-Air  Missile  (AMRAAM)  testing  at  the  Guided 
Weapons  Evaluation  Facility  (GWEF)  at  Eglin  AFB  used  M&S  to  simulate  between  16,000  and 
17,000  missile  software  and  hardware  tests.  This  testing  was  not  feasible  using  live  firings  due 
to  the  high  cost  (about  $3M  for  each  missile).  The  needed  data  could  not  be  obtained  unless 
M&S  was  used. 

f.  Navy  in-water  torpedo  testing  can  cost  from  $50,000  to  $80,000  per  firing,  while 
the  used  of  simulation  can  allow  100  to  300  tests  to  be  run  for  the  same  price. 

g.  Navy  live-fire  missile  testing  produces  fewer  than  1 00  data  points  on  average,  and 
hundreds  of  thousands  of  such  data  points  are  needed  to  adequately  describe  missile  performance 
envelopes.  Since  fewer  than  25  valid  live  firings  are  usually  affordable,  the  only  alternative  is 
the  use  of  simulation. 

h.  An  analysis  of  the  HWIL  facility  at  Point  Mugu,  CA  for  AMRAAM  testing 
claims  a  cost  avoidance  of  $2,500  million  a  year  (assuming  that  8,400  (the  number  of  simulated 
firings)  live  firings  per  year  was  feasible).  The  same  analysis  reveals  that  the  Weapons  Software 
Support  Facility  (WSSF)  at  China  Lake  provides  a  cost  avoidance  of  about  $10  million  a  year. 
WSSF  is  used  for  integration,  checkout,  and  V&V  of  avionics  software  with  actual  avionics 
hardware  operating  as  a  total  aircraft  system.  The  conclusion  is  that  it  is  impossible  to  rigorously 
test  systems  as  complex  as  F/A-18  avionics  without  this  type  of  facility,  not  only  from  the  point 
of  view  of  cost,  but  also  with  regard  to  safety,  accuracy,  environmental  concerns,  security/stealth, 
training,  and  other  factors. 

i.  During  SIMNET/AIRNET  Developmental  and  Operational  Testing  for  Army 
helicopters,  two  Non-Line  of  Sight  (NLOS)  simulators  were  built  and  compared  with  captive 
flight  and  actual  missile  firing  tests.  The  results  showed  that  total  costs  for  the  necessary  testing 
was  about  $6.65M  for  captive  flight,  $8.8M  for  live  missile  firing,  and  $2M  for  the  simulators. 
The  simulators  had  major  advantages  in  the  number  of  trials  that  were  completed,  and  the 
reduced  time  and  number  of  test  personnel  required. 

j.  The  US  Army  Missile  Command  (MICOM),  RD&E  Center,  Advanced 
Simulation  Center  Hardware-in-the-Loop  Simulation  Facilities,  claim  total  cost 
savings/avoidance  that  exceed  $270M. 

k.  US  Army  TECOM  has  documented  the  following  benefits  derived  from  M&S,  for 
ground  weapon  systems: 


•  The  Firing  Impulse  Simulator  (FIS)  simulates  the  recoil/trunnion  loads  and  ballistic 
shock  effects  for  tank  and  howitzer  cannons.  The  investment  cost  was  almost  $7M,  but 
the  FIS  saves  about  $2,000  per  round  or  $23M  in  cost  avoidance  for  a  typical  trunnion 
bearing  test.  There  are  also  savings  from  reduced  personnel  (13  to  4),  time  savings,  and 
reduced  environmental  problems  in  terms  of  noise,  blast,  and  toxic  fumes  for  each  test. 

•  The  Moving  Target  Simulator  (MTS)  assesses  the  ability  of  tank  and  other  weapon 
crews  to  track  and  fire  on  simulated  maneuvering  targets  (represented  by  projected  laser 
spots).  The  MTS  saves  about  $1 .5M  per  year  compared  to  field  tests,  and  also 
significantly  reduces  time  spent  on  test,  redesign,  and  retest. 

•  Bridge  durability  testing  was  once  done  by  3,000  actual  crossings  over  a  period  of  12 
weeks  and  at  a  cost  of  about  $325,000.  Such  tests  are  now  conducted  using  a  mix  of 
actual  and  simulated  crossings  based  on  instrumented  degradation.  The  new  method 
reduces  cost  to  about  $1 10,000  and  time  to  about  9  weeks. 

•  Vibration  Test  Facility  (VTF)  shaker  tables  can  now  replace  actual  miles  driven  by 
combat  vehicles  to  determine  the  effects  of  vibration.  This  saves  costly  field  testing,  as 
each  tank  hour  of  driving  costs  about  $1,200. 

•  Target  Acquisition  Model  Improvement  Program  (TAMIP)  provides  an  objective 
means  for  comparing  the  vulnerability  of  vehicles  to  detection  on  the  battlefield.  The 
models  quantify  the  value  added  when  vehicle  signatures  are  reduced.  Included  in 
TAMIP  improvement  are  millimeter  wave,  infrared,  visible,  and  acoustic  models. 

l.  The  AFFTC  at  Edwards  AFB,  CA,  primarily  conducts  developmental  flight  tests 
to  clear  aircraft  and  electronic  warfare  (EW)  systems  for  operational  testing.  Benefits  are 
significant,  but  hard  to  quantify: 

•  Immediate  cost  savings  from  M&S  use  in  flight  test  are  not  significant  compared  to 
the  long  term  (cost  avoidance)  savings  to  the  program  from  more  effective  and  timely 
elimination  of  problems,  especially  before  fielding  the  aircraft  (after  which  changes 
become  very  expensive). 

•  M&S  tools  are  essential  to  investigating  issues  of  flight  safety  before  live  range 
testing  with  humans. 

•  In  recent  years  using  M&S  tools  at  Edwards,  there  were  no  major  problems 
encountered  in  live  flights  that  were  not  identified  early  in  pre-test  simulations  by  those 
tools. 

•  M&S  helps  to  focus  the  available  test  time  much  more  productively  and  reduces  risk. 

m.  NAWCWPNS,  China  Lake,  CA,  is  the  Navy’s  full  spectrum  RDT&E  and  in- 
service  engineering  center  for  weapons  systems  associated  with  air  warfare  (except 
antisubmarine  warfare  systems),  missiles  and  missile  subsystems,  aircraft  weapons  integration, 
and  assigned  airborne  EW  systems.  A  range  of  facilities  within  the  NAWCWPNS  provide 
important  data  in  the  areas  of  target  signatures,  acquisition  and  tracking,  end-game  geometry, 
weapon  lethality,  proximity  sensor  detection  characteristics,  target  resistance  to  fusing,  avionics 
integration,  pilot  interactions,  radar  and  sensor  integration,  GPS  integration,  situational 
awareness,  rules  of  engagement,  radio  frequency  (RF)  and  infrared  (IR)  signatures,  and  other 
types  of  data.  Identified  benefits  include: 


•  M&S  is  an  enabling  tool  without  which  much  of  the  acquisition  process  cannot  be 
accomplished  as  it  allows  understanding  of  engineering  and  operational  issues  including 
difficult  systems  integration  issues;  assessment  of  scenarios  which  are  impossible  to 
actually  test;  and  evaluation  where  security  considerations  preclude  ‘open  air’  testing. 

•  M&S  avoids  the  cost  of  some  live  fire  test  failures  which  typically  result  in  program 
schedule  slips  and  unanticipated  costs,  or  high  visibility  failures  which  threaten  life  as 
well  as  program  viability,  and  other  aspects  of  risk. 

•  M&S  avoids  Software  Support  Activities  (SSA)  startup  costs  after  production  ends. 

•  M&S  reduces  costs  by  focusing  test  firings. 

n.  Sikorsky  projects  that  for  the  Comanche  helicopter  program  they  will  save  $673M 
through  the  use  of  virtual  prototyping  over  actual  flight  test  hours  involving  crew  station  and 
flight  controls  design,  major  equipment  integration,  air  worthiness  qualification  and  training. 

o.  A  vulnerability  assessment  was  required  for  each  of  the  15  primary  targets  for  the 
Joint-Service  Standoff  Weapon  (JSOW).  The  Navy  conducted  full-scale  tests  on  only  three  of 
the  targets  to  validate  and  calibrate  M&S  tools.  The  tools  were  then  used  to  conduct  the  analyses 
of  the  remaining  12  targets  without  further  full-scale  tests.  This  approach  saved  more  than 
$2.5M. 


p.  At  AEDC,  M&S  has  been  used  to  help  lower  the  cost  of  testing  to  the  customer. 
Average  time  in  the  PWT-16T  wind  tunnel  has  decreased  from  six  weeks  to  3-4  days. 

q.  At  Eglin  AFB,  use  of  PRIMES  (Preflight  Integration  of  Munitions  and  Electronic 
Systems)  ground  simulation  led  to  a  35%  reduction  in  cost  and  a  300%  increase  in  data  capture 
during  a  recent  flight  test  program  of  the  APG-63  radar. 

r.  The  Patuxent  River  Naval  Air  Warfare  Center  Aircraft  Division  (NAWCAD)  used 
state-of  -the-art  simulation  and  ground  test  capabilities  to  reduce  flight  test  hours  and  costs  by  a 
third  while  evaluating  ALQ-99  receivers  and  ALQ-149  communications  countermeasures 
equipment  on  board  the  EA-6B  aircraft. 

s.  The  F-16  program  has  used  AFFTC  M&S  facilities  to  evaluate  avionics  and 
electronic  suites  since  1990  as  a  replacement  for  and  complement  to  flight  test  programs. 
Approximately  $40.8  million  have  been  saved  on  flight  tests  with  this  approach. 

t.  M&S  tools  were  used  to  test  the  F-1 5  radar,  providing  three  times  the  data  at  a 
third  of  the  cost  compared  to  open  air  range  testing.  A  small  number  of  flight  tests  were  used  to 
validate  the  results. 

E.  Manufacturing  Process 

Emphasis  and  investment  in  automation  has  improved  the  efficiencies  of  the 
manufacturing  process,  but  its  lack  of  integration  with  other  development  processes  is  the  root 


problem.  Particularly  in  the  defense  industry  where  emphasis  is  on  design  and  innovative 
technology,  manufacturing  has  suffered  in  priority  and  emphasis. 

1 .  Tools  in  the  Evolving  Manufacturing  Process 

The  primary  contribution  of  emerging  M&S  tools  is  not  in  improved  manufacturing 
technologies,  but  rather  in  bringing  manufacturing  expertise  to  the  design  processes  so  that  the 
final  design  is  more  manufacturable.  Improved  manufacturability  offers  significant  potential 
payoff 

In  addition,  M&S  tools  can  assist  the  manufacturing  team  in  designing  the  manufacturing 
process  for  a  new  system  just  as  the  design  team  is  developing  the  design.  The  equipment,  work 
flow,  and  overall  process  for  manufacturing  can  now  be  developed  and  analyzed  in  a  virtual 
enviroiunent  with  high  confidence  in  the  results. 

2.  Metrics. 

-  number  of  available  options  to  improve  cost,  schedule,  performance,  etc. 

-  reduction  in  number  of  manufacturing  process  steps  and  time 

-  better  part  and  assembly  fit  resulting  in  less  need  for  rework 

-  reduced  labor  costs  including  fewer  meetings  and  data  submittals 

-  reduced  amount  of  scrap  and  waste  material 

3.  Benefits  of  M&S. 

a.  The  Comanche  program  used  the  Computer  Aided  Three-Dimensional  Interactive 

Simulation  (CATIA)  package  to  improve  on  the  traditional  design  and  manufacturing  processes. 
For  joining  the  fuselage  and  tailboom,  this  facilitated  a  two-step  tooling  process  versus  an  eight- 
step  process  as  experienced  in  the  CH-53  program  without  use  of  CATIA.  It  also  achieved  95% 
first  time  fit  versus  35%  in  previous  processes,  and  reduction  of  labor  cost  for  the  joining  from 
19  man-years  to  one  man-month. 

b.  The  Joint  Strike  Fighter  (JSF)  PM  Office  and  McDonnell  Douglas  conducted  a 
six-month  side-by-side  comparison  of  traditional  versus  virtual  manufacturing  (VM),  involving 
redesign  of  the  F-1 5E  mid-fuselage  airframe  former.  Benefits  attributed  to  the  VM  approach 
were: 

•  33%  reduction  in  design  release  time, 

•  27%  reduction  in  cost, 

•  1 9%  reduction  in  manufacturing  cycle  time, 

•  20%  reduction  in  factory  floor  space  use, 

•  a  higher  quality  production  part  including  24%  reduction  in  parts  count  and  78% 
reduction  in  fasteners  required  for  assembly. 


The  JSF  PM  Office  projects  the  benefits  of  VM  offer  a  potential  savings  of  up  to  3%  of 
life  cycle  costs  which  could  equate  to  $5  billion. 

c.  Through  computer-based  concurrent  engineering  and  improved  communications 
within  and  among  design  teams,  Intel  Corporation  has  reduced  the  time  for  hardware  from 
design-to-sample  in  half,  even  though  product  complexity  doubled.  In  addition,  the  company  has 
achieved  a  95%  success  rate  on  the  first  silicon  fabrication  of  new  products. 

d.  For  the  Army’s  Flexible  Computer  Integrated  Manufacturing  (FCIM)  program, 
the  Electronics  Module  communications  end  items  effort  has  resulted  in: 

•  66%  reduction  in  cycle  time, 

•  $3  million  in  cost  savings, 

•  $3.8  million  in  cost  avoidance. 

e.  Electronically  integrated  data  among  several  departments  permitted  Ford  Motor 
Company  to  increase  its  quality  such  that  there  was  a  10-15%  cost  reduction  and  a  14%  reduction 
in  time  for  sheet  metal  production.  Currently,  Ford  is  on  track  toward  its  goal  of  90%  reduction 
in  the  number  of  prototype  manufaeturing  models  that  it  must  build. 

f  The  use  of  integrated  CAD  systems  by  Northrop  led  to  a  first-time,  error-free 
physical  mock-up  of  many  B-2  sections.  Use  of  CAD  also  assisted  in  achieved  first-time  correct 
tube  bends  for  expensive  titanium  electronic  cable  conduits. 

g.  Boeing  Corporation,  using  new  M&S  tools  and  processes  on  the  777  aircraft,  was 
able  to  make  necessary  tooling  changes  with  only  two  engineers  instead  of  40  engineers  required 
for  previous  aircraft. 

h.  Boeing  reports  that  their  747  required  over  10,000  shims  to  compensate  for  ill- 
fitting  parts  while  the  777  requires  fewer  than  50  shims.  After  leaving  the  assembly  line,  the  747 
required  many  hundreds  of  workers  to  complete  unfinished  tasks,  while  the  777  required  only  a 
few  workers. 

i.  Boeing  reports  that  scrap  was  reduced  by  30%  on  the  777  compared  to  the  747. 
Rework  was  reduced  from  30%  on  the  747  down  to  3%  on  the  777. 

,  j .  The  JSF  program  reconfigured  one  component  that  was  initially  comprised  of  250 
parts  into  a  design  of  only  25  parts  which  significantly  reduced  manufacturing  and  support  costs. 

F.  Deployment  and  Support  Process 

Operation  and  support  (O&S)  of  a  deployed  system  typically  requires  a  majority  of  the 
total  life  cycle  costs  of  the  system.  The  cost  of  maintaining  the  system  is  a  function  of  many 
factors  including  the  maintenance  time  required,  and  the  production  and  storage  costs  of  repair 


parts.  Too  often  these  O&S  eosts  have  not  been  adequately  considered  by  the  design  team  which 
is  driven  by  operational  performance  requirements. 

1 .  Tools  in  the  Evolving  Deployment  and  Support  Process. 

As  in  previous  phases,  the  significant  change  made  possible  by  the  use  of  M&S  tools  is  to 
integrate  these  O&S  functions  into  the  total  system  development  process.  That  primarily  means 
considering  the  implications  of  these  functions  on  the  concept  which  is  selected  and  the  design 
which  is  developed  to  satisfy  operational  requirements.  Not  only  can  operational  use  be 
evaluated  during  the  design  stages  to  minimize  the  subsequent  necessity  for  modifications  to  the 
fielded  systems,  but  the  support  requirements  for  those  systems  can  be  better  analyzed  during  the 
design  stage  to  lessen  the  support  burden  and  thus  the  total  life  cycle  costs  of  the  system. 

2.  Metrics. 

-  time  to  evaluate  Operations  and  Support  (O&S)  costs  and  issues 

-  time  to  analyze/create  O&S  requirements  such  as  planning  documents 

-  amount  of  data  stored  and  accessed 

-  numberoflegacy  designs,  products,  or  tools  re-used 

3.  Benefits  of  M&S. 

a.  The  Navy’s  Smart  Product  Model  (SPM)  is  designed  to  support  all  phases  and 
functions  of  acquisition  including  O&S  needs.  For  the  NSSN  program,  early  analysis  of  form,  fit 
and  function  integrates  representatives  from  the  fleet  in  order  to  ensure  they  can  provide  input  to 
how  design  and  manufacturing  will  impact  future  maintenance  and  support  activities. 

b.  Lockheed  Martin  Corporation  indicates  that  M&S  support  in  the  areas  of 
supportability  evaluation  will  reduce  maintenance  man-hours  by  up  to  a  factor  of  three. 

c.  Electronic  integration  by  Northrop  permitted  the  reduction  of  provisioning  list 
release  time  from  six  months  to  60  minutes. 

d.  For  the  McDonnell  Douglas  AH-64D  Longbow  Apache  program,  the  Engineering 
Development  Simulator  (EDS)  was  initially  used  for  the  source  selection  effort,  then  was  further 
developed  and  verified  and  validated  to  support  development,  prototype  production,  and  testing. 
The  EDS  or  legacy  systems  built  upon  the  EDS  such  as  the  LCT  (Longbow  Crew  Trainer)  are 
now  used  for  aircraft  training.  The  MAVWEST  (Multiplex  Armament  Visionics,  Weapons  and 
Electrical  System  Trainer)  uses  hardware  and  software  developed  for  the  LCT  and  other  early 
legacy  systems.  MAVWEST  is  used  for  high  complexity  maintenance  training  for  a  range  of 
systems  including  fire  control  radar,  armament,  navigation,  and  coimmmications. 

e.  With  the  Navy’s  focus  on  life  cycle  maintenance,  new  M&S  tools  have  helped  to 
reduce  the  standard  parts  list  from  about  95,000  for  the  Seawolf  class  submarine  to  about  16,000 
for  the  NSSN. 


f.  A  study  of  the  impact  of  a  common  data  management,  storage,  retrieval  and 

exchange  service  for  transferring  in  a  standard  digital  format  all  contractor  design  and 
manufacturing  data  among  the  Air  Force  and  its  B2  subcontractors  found  significant  savings. 
This  study,  “CALS  Contractor  Integrated  Technical  Information  Service  (CITIS):  Business  Case 
Feasibility  Study,”  determined  that  the  CITIS  would  lead  to: 

•  50%  reduction  in  attendees  at  meetings  between  contractors  and  the  Air  Force, 

•  5.4%  reduction  in  the  total  B2  spare  parts  dollars, 

•  23%  reduction  in  modification  lead  time, 

•  1 .8%  increase  in  the  average  availability  of  the  aircraft  fleet,  and 

•  90%  reduction  in  the  contractor  data  submittals. 

The  total  estimated  cost  savings  ranged  from  a  minimum  of  $536M  to  a 
maximum  of  $894M,  for  investments  that  ranged  from  $9M  to  $30M. 


IV.  CHALLENGES  AND  OPPORTUNITIES 


A.  Introduction 

While  the  primary  objective  of  this  study  was  to  quantify  the  value  of  M&S  in  the  DoD 
acquisition  process,  and  that  was  addressed  in  Chapter  III,  it  was  also  recognized  that  the  same 
research  would  surface  data  on  challenges  and  opportunities  that  organizations  and  programs 
were  encountering  in  effectively  utilizing  M&S  tools  within  that  process. 

Not  surprisingly,  the  challenges  that  are  being  encountered  by  users  of  the  tools  are  far 
easier  to  capture  than  quantifying  the  value  of  M&S  tools  in  the  acquisition  process.  Every 
organization  has  a  list  of  problems  that  need  to  be  resolved,  and  many  of  them  have  common 
roots.  It  is  beyond  the  scope  of  this  study  to  develop  a  complete  list  of  issues,  the  solution  to 
which  would  result  in  the  optimal  use  of  M&S  tools  to  seamlessly  enhance  the  acquisition 
process,  but  key  issues  are  identified  and  discussed  in  the  following  pages. 

The  technical  challenges  identified  in  this  chapter  are  not  beyond  the  ability  of  improving 
technology  to  resolve,  although  prudent  investments  or  proper  encouragement  might  well  speed 
and  enhance  the  usefulness  of  the  technical  solutions.  Cultural  and  managerial  issues  are  often 
more  difficult  to  overcome,  but  because  they  are  often  the  product  of  current  incentives,  both 
negative  and  positive,  built  into  the  acquisition  process,  there  is  reason  to  believe  that  effective 
solutions  are  available. 

In  addition,  there  are  many  noteworthy  initiatives  currently  underway  which  offer 
promise  for  improving  the  technologies  and  processes  which  support  DoD  acquisition.  Some  of 
those  are  summarized  at  the  end  of  this  chapter. 

The  intent  of  this  chapter  is  to  identify  and  describe  or  explain  those  challenges  and 
opportunities  to  the  extent  necessary  for  the  reader  to  gain  understanding.  Specific  findings  and 
recommendations  are  deferred  to  Chapter  V. 

B.  Challenges 

The  challenges  which  were  identified  in  this  research  have  been  organized  into  three 
groups:  technical,  cultural,  and  managerial.  While  these  groupings  and  their  titles  are  somewhat 
arbitrary,  and  some  challenges  could  be  classified  into  more  than  one  group,  it  emphasizes  the 
fact  that  not  all  of  the  problems  in  effectively  using  M&S  tools  in  the  acquisition  process  are 
technical  in  nature.  In  fact,  the  majority  of  the  challenges  are  caused  by  the  processes  in  which 
the  tools  are  employed,  and  by  the  experience  and  understanding  of  management  in  using  these 
tools. 


While  technical  issues  are  a  reasonably  obvious  category,  the  difference  between  cultural 
and  managerial  issues  is  somewhat  less  distinct.  The  general  distinction  in  this  chapter  is  that 
cultural  issues  are  related  to  work  processes,  worker  incentives,  and  education  and  training. 


Managerial  issues  are  related  to  official  guidance,  management  direction,  and  organizational 
design. 


1.  Technical 

a.  Interoperability  of  M&S  Tools 

(1)  Issue:  A  data  model  of  a  system  does  not  operate  seamlessly  with  the 
M&S  tools  across  the  spectrum  of  functional  areas  in  the  acquisition  process. 

(2)  Discussion:  As  described  in  previous  chapters,  these  M&S  tools  have 
been  incorporated  piecemeal  into  functional  areas  of  the  acquisition  process.  Historically,  new 
data  descriptions  of  the  system  were  developed  to  support  each  M&S  tool.  The  great  advantage 
of  using  the  same  (evolving)  data  description  throughout  the  development  and  fielding  of  the 
system  is  that  it  has  led  to  increased  effort  to  package  or  translate  the  data  description  so  that  it  is 
usable  by  many  (if  not  all)  of  the  M&S  tools  in  the  other  functional  areas.  While  this  is 
technically  possible  in  most  cases,  it  is  unduly  expensive  and  time  consuming.  Many  translators 
have  been  developed  between  various  types  of  M&S  tools,  but,  as  these  tools  are  upgraded  or 
enhanced,  the  translators  often  need  to  be  significantly  reworked  at  great  expense  of  people  and 
time.  The  increased  use  of  commercial-off-the-shelf  (COTS)  CAD/CAM/CAE  packages  has 
both  helped  and  exacerbated  the  situation  because  there  are  still  several  different  data  standards 
in  use  by  those  developers. 

(3)  Examples: 

•  TARDEC  often  requires  manual  reprogramming  to  go  from  one  functional 
area  to  another,  e.g.,  from  design  to  performance  modeling,  to  man-in-the-loop,  to  production 
analysis. 

•  TARDEC  has  developed  translators  from  CAD  design  packages  (Intergraph) 
to  their  performance  M&S  tools,  but  updates  to  Intergraph  then  require  extensive  work  to  update 
the  translators. 

•  TECOM  notes  that  one  of  their  biggest  challenges  is  the  development  of 
standard  interfaces  between  M&S  tools,  especially  the  output  of  CAD  data  directly  into  dynamic 
performance  M&S  tools. 

•  The  SC-21  Program  is  concerned  that  there  are  too  many  imcertainties 
regarding  future  protocols,  standards,  architecture,  and  infrastructure  for  M&S  tools  and  data.  It 
is  a  detriment  to  their  ability  to  plan  and  develop  an  effective  integrated  M&S  approach  to 
supporting  the  acquisition  process. 

•  The  AFATDS  Program  notes  that  most  digital  systems  and  data  are  not  joint 
or  interoperable.  Standardized  data,  below  the  architecture/infrastructure  level,  is  needed  to 
make  M&S  tools  work  easily  with  each  other. 

•  AEDC  has  the  same  concerns,  but  does  have  some  engineering  level  M&S 
tools  that  integrate  well  with  each  other  and  with  certain  campaign  level  tools.  They  do  a  great 
deal  of  work  directly  with  industry  and  need  the  flexibility  to  work  with  various  data  packages. 


•  LPD- 1 7  program  asked  for  design  databases  from  the  six  competing 
contractor  teams.  All  six  used  different  database  systems  and  none  were  interoperable  with  the 
system  in  use  at  the  Program  Office. 

•  The  National  Automotive  Center  (NAC),  a  TARDEC  consortia  with  industry, 
is  attempting  to  develop  a  collaborative  program  with  the  Big  Three  US  automobile 
manufacturers  and  the  major  ground  vehicle  manufacturers  to  implement  an  Automotive  Product 
Development  Framework  to  help  solve  this  interoperability  challenge. 

•  Ford  Motor  Company  has  purchased  and  implemented  a  standard 
CAD/CAM/CAE  approach  company-wide  to  overcome  this  challenge. 

•  Deneb  reports  that  corporations  which  have  made  a  heavy  investment  in 
internal  proprietary  M&S  development  consider  it  a  competitive  advantage  and  become  resistant 
to  COTS  approaches. 

b.  Availability  of  Data  Descriptions 

(1)  Issue:  Adequate  data  descriptions  of  current  weapon  systems  are  often  not 
available  for  M&S  support  of  the  system  upgrade/product  improvement  process. 

(2)  Discussion:  Data  descriptions  of  currently  fielded  weapon  systems  present 
a  special  challenge  because  of  the  time  that  has  elapsed  since  their  original  design  and 
manufacture.  Differences  in  contract  requirements,  tools  to  develop  and  maintain  data 
descriptions,  and  the  changing  contractor  environment  of  mergers  and  focus  on  DoD  business, 
result  in  a  potential  tangle  of  difficulties  as  those  systems  are  identified  for  major  upgrades 
instead  of  designing  a  new  system  to  replace  them.  Lack  of  availability  of  an  adequate  data 
description  of  the  current  system  will  add  significant  cost  to  the  design  and  development  of  any 
upgrade  as  well  as  limiting  full  contractor  competition  in  that  process. 

(3)  Examples: 

•  TARDEC  is  now  supporting  the  upgrade  of  many  currently  fielded  vehicle 
systems,  and  expects  to  support  many  more  in  the  future,  for  which  only  the  original  contractor 
has  a  data  description.  It  is  very  expensive  for  TARDEC  to  develop  an  adequate,  vector-based 
description  of  the  system  for  their  use  in  M&S  dynamic  analysis  of  design  concepts  and 
tradeoffs. 

•  AFFTC  Edwards  experiences  situations  where  no  system  model  exists,  or  the 
Edwards  staff  is  unable  to  understand  or  use  the  model  provided. 

c.  Securitv/Sensitivitv  of  Data 

(1)  Issue:  Situations  exist  where  system  data  must  be  kept  separate  from 

competing  contractors  or  secure  from  non-US  subcontractors  while  still  being  consistent  and 
available  to  the  US  Government. 

(2)  Discussion:  Early  involvement  of  multiple  competing  contractors  in  the 

design  and  development  process  of  a  new  weapon  system  is  highly  desirable,  if  not  essential. 
However,  the  evolution  toward  a  more  integrated  process  across  a  distributed  environment  using 


M&S  tools  and  a  common  data  description  of  the  new  system  complicates  the  need  to  provide 
adequate  protection  for  proprietary  information  and  processes.  There  are  also  frequent  situations 
where  a  US  prime  contractor  needs  to  share  M&S  tools  and  processes  with  non-US  sub¬ 
contractors,  but  certain  data  must  be  protected  in  the  process. 

(3)  Examples: 

•  SC-21  has  a  need  to  keep  a  proprietary  security  wall  between  the  M&S  data  of 
competing  contractor  teams  while  encouraging  their  frill  participation  in  the  design  and 
development  process  utilizing  a  common  database. 

•  NAWCWPNS  China  Lake  reports  programs  with  non-US  subcontractors 
where  special  “sanitized”  versions  of  system  models  and  associated  data  were  required  to  support 
their  involvement  in  the  system  development. 

d.  Physics-based  M&S 

(1)  Issue:  Many  M&S  tools  are  based  on  empirical  data  rather  than  physical 

principles. 

(2)  Discussion:  An  important  aspect  of  gaining  acceptance  of  the  output  of 
M&S  tools  is  to  base  them  on  proven  principles  of  physics,  validated  by  experimental  data.  The 
physics  of  many  processes  essential  to  DoD  weapons  systems  are  not  yet  adequately  imderstood, 
e.g.,  secondary  and  tertiary  effects  of  ballistic  weapons  impacting  a  target,  synergistic  effects, 
friction,  and  bending. 

(3)  Examples: 

•  TARDEC  has  needs  for  better  representation  of  dirty  environments  such  as 
mud,  dust,  and  smoke. 

•  The  physics  of  many  ballistics  events  are  not  yet  well  understood,  so  M&S 
tools  supporting  live  fire  testing  at  TECOM  are  typically  empirically-based. 

e.  Hardware  and  Software  Limitations 


(1)  Issue:  Limitations  still  exist  with  available  hardware  and  software  to 
support  emerging  requirements  of  M&S  in  the  acquisition  process. 

(2)  Discussion:  Although  the  availability  of  significantly  more  powerful 
computer  hardware  and  software  have  been  drivers  in  the  increased  incorporation  of  M&S  tools 
in  the  acquisition  process,  requirements  continue  to  emerge  for  faster,  larger,  and  more  powerful 
support  tools. 


(3)  Examples: 

•  NSSN  Program  Office  (PO)  workstations  are  not  adequate  for  use  with  their 
large,  dynamic  databases  which  need  more  than  2  gigabytes  of  RAM.  They  also  note  that 
software  developers  are  focusing  on  updates  to  workstation  applications  instead  of  mainframe 


applications,  because  of  the  significantly  larger  market.  This  may  result  in  future  computational 
limitations. 

•  Deneb  Robotics,  Inc.  (major  producer  of  3D  graphics-based  simulation)  has 
developed  for  Electric  Boat  Company  what  they  claim  is  the  world’s  largest  virtual  prototype 
model.  It  has  one  million  polygons  and  requires  two  gigabytes  of  RAM. 

f.  Variable  Resolution 

(1)  Issue:  Consistent  system  data  descriptions  are  needed  for  both  high  and 
low  resolution  M&S  tools. 

(2)  Discussion:  Typical  engineering  level  M&S  tools  are  very  high 
resolution,  and  CAD  descriptions  of  systems  and  subsystems  are  developed  to  provide  needed 
data.  Many  dynamic  performance  models  may  only  accommodate  a  lower  level  of  resolution 
thus  causing  loss  of  fidelity  and  attributes  of  the  original  CAD  developed  model  which  could 
skew  final  results.  Force  structure  and  affordability  analyses  may  require  very  low  level  data 
resolution.  In  all  cases,  the  representation,  at  both  higher  and  lower  resolutions,  needs  to  remain 
consistent. 

(3)  Examples: 

•  SC-21  Program  has  requirements  for  system  data  to  support  varying  resolution 

M&S. 

•  TARDEC  has  requirements  for  system  data  to  support  varying  resolution 

M&S. 

2.  Cultural 

a.  Acquisition  Processes 

(1)  Issue:  New  processes  are  required  to  realize  the  full  benefits  of  using 
M&S  tools  in  the  total  acquisition  process. 

(2)  Discussion:  Optimal  use  of  M&S  tools  require  the  development  of  new 
processes  with  a  culture  of  teamwork  and  openness. 

(3)  Examples: 

•  AFFTC  Edwards  has  experienced  program  managers  wanting  a  “fly-fix-fly” 
approach  instead  of  a  “predict-test-compare”  approach  which  they  believe  is  more  cost  effective. 

•  Deneb  Robotics  cautions  its  potential  customers  not  to  buy  their  software 
unless  they  are  committed  to  changing  the  way  they  do  business  toward  a  genuine  IPT  approach. 
They  stress  that  organizational  leaders  must  emphasize  change,  department  heads  must  become 
coaches,  and  new  approaches  must  be  developed  to  maintain  core  competencies  that  will  be 
matrixed  to  tasks. 


•  TARDEC  notes  that  current  engineering  practices  are  still  based  on  “old” 
rules-of-thumb  which  were  developed  over  many  years.  The  value  of  these  practices  needs  to  be 
retained,  but  adapted  to  new  ways  of  doing  business. 

•  The  automobile  industry  experiences  resistance  to  change  from  smaller 
suppliers  who  cannot  afford  the  M&S  tools  or  training  required  with  the  new  processes. 

b.  Incentives  for  M&S  Use 

(1)  Issue:  The  use  of  M&S  tools  and  processes  is  not  appropriately 

incentivized. 


(2)  Discussion:  Incentives  for  a  DoD  weapons  system  PM  are  tied  to  an  old 
acquisition  process.  This  process  is  designed  to  stop  problems  and  failures  that  are  not  nearly  as 
relevant  in  today’s  environment.  Although  almost  every  aspect  of  the  acquisition  process  is 
undergoing  change,  and  new  M&S  tools  and  processes  are  now  available  to  support  better 
quality  at  a  lower  cost,  the  incentives  for  necessary  up-front  investments  in  technology  and 
infrastructure  are  not  yet  present. 


(3)  Examples: 

•  Many  PM  offices  have  Initial  Product  Testing  as  an  early  goal  which 
stimulates  funding  release  for  production  and  fielding.  Up  front  time  and  funding  invested  in 
M&S  tools  and  processes  for  a  better  product  or  lower  risk  is  perceived  as  detracting  from  their 
incentive  for  early  fielding. 

•  AFFTC  Edwards  experiences  some  programs  that  don’t  accept  the  value  of 
M&S  tools  and  “drive  the  program  pace”  by  schedule,  cost,  or  similar  pressures. 


c.  M&S  Workforce 

(1)  Issue:  Availability  of  trained  personnel  to  support  the  development  and 

use  of  M&S  tools  and  processes. 

(2)  Discussion:  In  an  era  of  significant  Government  downsizing,  it  is  typical 
for  the  more  recently  hired  employees  to  be  the  first  to  depart.  In  the  fields  of  computer 
hardware  and  software  use  and  development,  it  is  these  more  recently  educated  workers  who 
possess  the  skills  and  education  required  to  implement  and  operate  the  technology.  In  addition  to 
the  pressures  to  down  size,  these  same  workers  are  in  demand  by  private  industry.  There  are 
concerns  about  the  availability  of  adequate  numbers  of  trained  personnel  in  Government  agencies 
and  acquisition  program  offices  to  support  the  development  and  use  of  M&S  tools  and  processes. 

(3)  Examples: 

•  TARDEC  suggests  that  individuals  or  organizations  which  use  an  M&S  tool 
should  be  “validated”  on  their  understanding  of  that  tool  before  results  are  accepted. 

•  AEDC  is  experiencing  difficulty  attracting  and  retaining  staff  in  many  areas 
requiring  skill  and  experience  in  the  use  of  emerging  M&S  tools  and  processes. 


Smaller  contractors  supporting  DoD  and  the  automobile  industry  have 
problems  training  personnel  to  operate  the  newer  M&S  tools  and  processes. 

d.  Acceptance  of  M&S 

(1)  Issue:  The  decision  ofhow  and  in  what  way  to  use  M&S  tools  and 

processes  is  often  based  on  inadequate  knowledge  and  experience. 

(2)  Discussion:  The  use  ofM&S  tools  and  processes  is  often  a  function  of  the 

experience  and  personality  of  the  key  program  decision  makers.  Their  level  of  familiarity  and 

comfort  with  these  newer  approaches  is  a  major  determinant  in  decisions  to  take  advantage  of 
these  tools. 


(3)  Examples: 

•  AFFTC  Edwards,  NAWCWPNS  China  Lake,  and  TARDEC  report  similar 

experiences  in  the  inconsistent  acceptance  of  M&S  tools  and  processes  by  different  program 
offices. 


3.  Managerial 

a.  OSD  and  Service  GniHanre 

(1)  Issue:  OSD  and  Service  guidance  fails  to  adequately  encourage  use  of 
M&S  tools  and  processes  in  support  of  the  acquisition  process. 

(2)  Discussion:  Although  the  DoD  acquisition  process  is  moving  towards  less 
mandatory  direction  and  requirements,  there  is  a  lack  of  knowledge  and  experience  in  how  to 
leverage  the  opportunities  presented  by  M&S  tools  and  processes  to  produce  a  better  weapon 
system  within  schedule  and  cost.  OSD  and  Service  guidance  which  helps  the  understanding  of 
how  and  when  to  incorporate  these  tools  would  produce  significant  dividends. 

(3)  Examples: 

•  There  is  very  little  guidance  in  DSMC  on  availability  of  tools,  and  how 
program  managers  can  capitalize  on  their  investments. 

•  The  Army  requires  program  managers  to  submit  an  SSP  for  ACAT I  and  II 
programs.  Even  with  speeifie  guidance,  this  has  been  perceived  as  another  bureaucratic 
requirement. 


b.  Ownership  of  Data 

(1)  Issue.  Lack  of  Government  rights  to  system  data  and  open  access  to  M&S 

tools  used  by  a  contractor,  limits  the  usefulness  and  value  of  M&S  in  support  of  the  process. 

(2)  Discussion:  This  remains  a  serious  issue  limiting  the  effectiveness  of 
M&S  tools  and  processes.  This  research  project  has  not  focused  specifically  on  this  issue  to 


determine  its  scope,  but  frequent  cases  were  identified  to  the  study  team  for  both  currently 
fielded  weapons  systems  as  well  as  those  under  development  where  this  lack  of  full  access  to 
system  data  descriptions  is  a  hindrance  to  effective  use  of  M&S  in  support  of  the  program.  Data 
rights  are  a  significant  issue  in  the  negotiations  for  a  development  contract  for  many  reasons 
other  than  M&S  use,  but  it  should  be  recognized  that  the  requirement  for  reusability  of  that  data 
by  other  M&S  tools  is  a  significant  element  in  the  success  of  the  program. 

(3)  Examples: 

•  AFFTC  Edwards  experiences  situations  where  the  contractor  considers  the 
model  proprietary  and  not  available  to  the  Government. 

•  Sikorsky  builds  and  owns  simulations  used  on  Comanche  and  other  programs 
which  are  not  available  to  the  Government. 

•  Javelin  program  has  experienced  concerns  about  proprietary  data  and  models. 
Contractors  have  been  encouraged  to  build  their  ovra  tools  which  then  become  proprietary. 

•  Mr.  Coyle,  DOT&E,  and  many  others  in  DoD  have  suggested  requiring  a 
system  model  as  part  of  the  submission  with  a  competitive  proposal.  TACOM  has  used  this 
approach  to  a  limited  degree,  but  legal  obstacles  have  severely  restricted  the  process.  Comanche 
Program  Office  used  “fly-offs”  between  simulators  as  part  of  their  Demonstration  and  Validation 
(DEMVAL)  award  evaluation. 

c.  VV&A  Requirements 

(1)  Issue:  The  product  of  M&S  tools  must  have  credibility  throughout  the 
acquisition  process. 

(2)  Discussion:  M&S  tools  have  little  value  in  the  acquisition  process  if  their 
contribution  is  not  accepted,  but  the  verification  and  validation  (V&V)  process  leading  to 
accreditation  is  expensive  and  not  well  understood. 

(3)  Examples: 

•  TARDEC  is  struggling  with  how  to  fund  needed  V&V  activities  of  M&S  tools 
which  are  needed  for  use  by  many  programs.  Individual  programs,  many  of  which  are  small,  are 
reluctant  to  carry  the  burden  of  funding  an  activity  needed  by  other  programs. 

•  AEDC  experienced  unclear  guidance  and  mixed  acceptance  of  their  V&V 
activities  from  Service  agencies  using  their  analyses. 

d.  Funding  Process 

(1)  Issue:  There  is  no  process  to  determine  appropriate  investment  in  M&S 
tools  and  processes  for  acquisition  programs. 

(2)  Discussion:  Funding  for  M&S  tools  and  processes  is  embedded  in  total 
program  funding  and  not  separately  identified.  Thus,  each  program  faces  issues  of  the  level  of 
investment  in  these  tools  and  proeesses  as  a  tradeoff  with  other  possible  investments  or 
expenditures  within  the  program. 


(3)  Examples: 

•  AFFTC  Edwards  is  often  not  able  to  update  models  with  live  test  data  at  the 
completion  of  testing  because  of  lack  of  ftmding  or  schedule  demands. 

•  TARDEC  is  struggling  with  the  reluctance  and  inability  of  individual  Army 
program  managers  to  make  up-front  M&S  investments. 

e.  Use  of  System  Models 

(1)  Issue:  There  is  no  control  mechanism  or  agency  to  insure  authoritative 
system  models  are  used  by  those  outside  of  the  system  program  office. 

(2)  Discussion:  The  general  availability  of  system  models  allows  other 
programs  or  organizations  to  misuse  that  data  to  support  programmatic  claims  or  decisions 
detrimental  to  the  original  system.  This  situation  causes  program  offices  to  overly  control  access 
to  their  system  models  to  avoid  misuse. 

(3)  Examples: 

•  TOMAHAWK  Program  Office  reports  use  of  unauthorized  representations  of 
their  system  model  by  outside  organizations. 

•  Multiple  Launch  Rocket  System  (MLRS)  claimed  that  representations  of  their 
system  were  not  blessed  by  the  Program  Office  when  played  in  higher  level  models. 

•  Navy  PEOs  have  expressed  concern  about  a  Navy  initiative  to  represent  their 
systems  in  a  common  frame  of  reference  which  would  make  them  subject  to  ‘trade-off  analyses. 

C.  Opportunities 

Industry  and  Government  have  similar  interests  and  are  cooperating  closely  in  many  areas 
pertaining  to  M&S  and  streamlined  acquisition.  In  reaction  to  global  or  national  competition  and 
other  pressures,  many  firms  have  modified  their  approach  to  product  development, 
manufacturing,  and  support.  In  many  cases  they  are  using  M&S  tools  and  processes  extensively. 
Many  of  these  firms  have  influenced  the  evolution  of  DoD  acquisition  policy,  especially  firms 
that  are  DoD  contractors,  and  are  forging  closer  alliances  through  teamwork  and  IPT/IPPD 
initiatives.  Most  large  system  contractors  (such  as  those  that  bid  for  DoD  ship,  aircraft,  tactical 
vehicle,  and  similar  contracts)  cannot  afford  mistakes  during  development,  thus  rely  increasingly 
on  the  advantages  provided  by  M&S. 

The  result  is  an  increase  in  number  and  influence  of  various  consortia,  cooperative 
research  and  development  agreements  (CRDAs),  and  dual-use  efforts.  These  trends  lead  to 
greater  cooperation  and  efficiency. 

An  example  is  the  National  Coordinating  Office  for  High  Performance  Computing  and 
Communications  (HPCC).  Under  the  HPCC  umbrella  is  the  Army  High  Performance 
Computing  Research  Center  (AHPCRC).  This  organization,  in  conjunction  with  the  Army 
Research  Lab  (ARL),  is  undergoing  a  transformation  to  a  concept  known  as  the  ‘federated 


laboratory’  which  has  promise  to  make  ARL  more  efficient.  “DoD  policy  is  to  use  commercial 
technology  wherever  possible,  to  out-source  even  technology  base  work,  when  the  private  sector 
has  the  lead  in  the  technology,  and  to  develop  technology  such  that  it  is  suitable  for  both  military 
and  civilian  applications  -  dual  use  -  thereby  obtaining  economies  of  scale.” 

Several  methods  for  conducting  joint  business  are  being  institutionalized.  They  include 
cooperative  agreements,  independent  research  and  development  (IR&D),  and  various  methods  of 
domestic  technology  transfer  such  as  CRD  As  and  PLAs  (Patent  Licensing  Agreements).  These 
will  allow  new  working  arrangements  while  minimizing  the  contractual  burdens  on  both  parties. 
Another  new  thrust  is  the  routine  exchange  of  scientists  and  engineers  between  industry  and 
Government. 

Following  are  descriptions  of  opportunities  in  developing  and  improving  simulation 
based  acquisition  by  various  programs  and  consortia  formed  to  jointly  address  common  issues  in 
this  area. 

1 .  DARPA’s  Simulation  Based  Design  (SBD) 

DARPA  plays  a  leading  role  in  a  number  of  M&S  programs.  A  key  project  is  SBD 
which  was  initiated  in  March  1993.  The  goal  of  the  project  is  to  ‘revolutionize  the  Acquisition 
Process  for  complex  military  and  commercial  products’  using  distributed,  collaborative  virtual 
development  environments.  SBD  is  multifaceted  in  that  it  looks  at  all  aspects  of  the  system 
acquisition  process,  from  mission  analysis,  through  design  and  logistics  considerations,  to 
manufacturing  and  cost/risk  analysis  phases.  The  vmderlying  belief  is  that  M&S  can  improve 
time  and  cost  for  all  areas  of  the  acquisition  process. 

Phase  I  of  the  program  has  been  completed.  As  ships  are  complex  and  costly  systems,  a 
ship  hull/mechemics/electrical  problem  was  selected  for  use  in  Phase  I.  The  challenge  was  to 
design  and  build  a  roll-on/roll-off  ship  that  could  transit  a  given  distance  faster  than  current 
ships.  First,  from  a  catalog  of  ‘product  models’  it  was  demonstrated  that  a  user  could  select  a 
new  synthetic  engine  product  model  and  introduce  it  into  a  3-D  CAD  model  of  the  ship. 

Through  associativity,  the  CAD  model  identified  ‘fouls.’  An  example  of  a  foul  might  be  that  the 
cooling  capacity  is  insufficient  for  the  new  engine.  The  designer  could  then  find  a  product  model 
of  a  new  cooler,  and  select  and  ‘install’  it  into  the  ship  model.  Another  foul  might  be  that  space 
is  insufficient.  Using  virtual  immersion,  the  designer  is  permitted  to  move  walls  and  explore  the 
impact  of  the  moves.  For  each  alteration,  a  manufacturing  analysis  model  is  used  to  make 
modifications  needed  to  the  sequence  in  which  the  ship  is  to  be  assembled. 

Phase  II  of  the  project,  a  $20M  a  year  program,  is  currently  underway.  The  contractor 
team  is  headed  by  Lockheed  and  includes  Boeing,  General  Dynamics,  and  Newport  News 
Shipbuilding.  This  project  will  test  SBD  as  a  basis  of  simulation  based  acquisition.  Their  initial 
test  is  scheduled  for  February  1997. 


2.  Other  DARPA  Programs 

The  MADE  (Manufacturing  Automation  and  Design  Engineering)  program  has  as  its 
long-term  objective  the  development,  integration,  and  demonstration  of  enabling  technologies  for 
design  of  complex  electro-mechanical  systems  that  will  lead  to  affordable  insertion  of  technology 
for  force  modernization.  The  technical  approach  is  to  leverage  existing  infrastructure  using  the 
WWW  as  the  integration  substrate,  and  to  develop  tools  that  incorporate  downstream  concerns 
into  early  design-stage  consideration.  MADE  includes  a  team  of  major  contractors  including 
industry,  tool  vendors,  and  universities.  It  is  a  four-year,  $47M  effort.  The  program  is  divided 
into  the  following  four  areas:  design  representation  and  conceptual  design,  integration 
architectures  and  multi-disciplinary  optimization,  design  exploration,  and  designer’s  interfacing. 
Expected  program  benefits  include  computer-based  engineering  tools  to  enable  geographically 
distributed  teams  to  meet  DoD  IPPD  requirements. 

The  AMS  (Affordable  Multi-Missile  Manufacturing)  is  funded  by  DARPA  for 
approximately  $100M  over  five  years.  The  goal  is  to  develop  and  demonstrate  innovative 
system  concepts,  and  design,  manufacturing  and  business  practices  that  can  substantially  reduce 
the  cost  of  the  various  DoD  missiles  and  smart  munitions.  Some  concepts  being  pursued  by 
AM3  for  cost  reduction  include:  multi-missile  component  technology,  IPPD  process  and  tools, 
flexible  manufacturing  systems,  enterprise  electronic  infrastructure,  and  streamlined  business 
practices. 

The  goal  of  the  RASSP  (Rapid  Prototyping  of  Application  Specific  Signal  Processors) 
program  is  to  dramatically  improve  the  design  process  for  complex  digital  systems.  RASSP  is  a 
$150M  DARP A/Tri-Service  initiative  that  started  about  two  years  ago  and  has  already 
implemented  the  first  RASSP  system  which  represents  a  significant  advance  over  today’s  state- 
of-the-art.  A  key  objective  is  to  reduce  the  total  product  development  time  by  a  factor  of  four  or 
greater,  while  improving  product  quality  and  reducing  life  cycle  cost.  Other  goals  are  to  field 
state-of-the-art  products  at  system  build  time,  and  permit  rapid  upgrade  to  the  system  throughout 
its  life  cycle.  RASSP  is  meeting  its  goals  through  a  combination  of  advanced  design  methods 
that  emphasize  virtual  prototyping,  concurrent  engineering,  and  design  reuse;  modular,  scaleable 
signal  processor  architectures;  and  a  comprehensive  support  base  of  electronic  design 
infrastructure.  The  infrastructure  includes  recent  updates  to  standards,  automation  tools,  and 
enterprise  integration  capabilities,  and  hardware  and  software  libraries.  The  program  also 
adopted  the  incremental  refinement,  or  ‘model  year,’  design  process  in  order  to  achieve  short 
development  schedules  (3  to  12  months),  continuous  improvement,  and  to  avoid  point  design 
solutions. 

DARPA  is  associated  with  a  wide  range  of  programs  intended  to  improve  manufacturing 
effectiveness.  Four  major  DARPA/DoD  initiatives  are:  the  Scaleable  Flexible  Manufacturing 
Program  that  focuses  on  electronics,  materials,  and  information  systems;  Technology  for 
Affordability  program;  Technology  Reinvestment  Program  (TRP);  and  Manufacturing  Science 
and  Technology  (ManTech)  Program. 


The  MatiTech  program  is  closely  associated  with  the  Manufacturing  Technology 
Information  Analysis  Center  (MTIAC),  one  of  a  series  of  organizations  chartered  by  DoD  to 
facilitate  use  of  existing  scientific  and  technical  information.  MTIAC  has  a  range  of  research 
interests  relevant  to  this  study,  such  as  Electronics  Processing  and  Fabrication,  Manufacturing 
and  Engineering  Systems  (includes  MADE ),  Advanced  Industrial  Practices  (includes  AMS 
program),  and  Metals  Processing  and  Fabrication.  The  ManTech  investment  strategy  is  to  apply 
small  investments  in  selected  areas  in  order  to  leverage  the  billions  of  dollars  that  commercial 
industry  is  investing  to  excel  in  world-class  competition.  Many  of  the  best  methods,  processes, 
and  capabilities  can  be  adopted  to  develop  and  produce  military  products.  The  ManTech  focus  is 
on  technology  demonstrations  and  technology  development.  Examples  of  current  ManTech 
demonstration  programs  include: 

-  Military  avionics  from  an  automotive  production  line  (50%  cost  savings) 

-  Military  aircraft  structure  using  commercial  practices  and  processes  (50%  cost 

reduction) 

-  Cost  and  cycle  time  reductions  for  machining  precision  aspherical  optical  lenses  (30- 

50%  reductions) 

-  Integrated  scheduling  throughout  a  mechanical  product  supply  chain  by  adapting 

commercial  tools  (90%  reduction  in  reschedule  time) 

3 .  ADP A’ s  Study  on  the  Application  of  M&S  to  the  Acquisition  of  Maj  or  Weapons 

Systems 

Industry  is  motivated  to  find  ways  to  improve  the  acquisition  process,  including  use  of 
M&S.  An  example  is  the  American  Defense  Preparedness  Association  (ADP A)  study  on  the 
application  of  M&S  to  the  acquisition  of  major  weapons  systems.  The  objective  was  to  present 
the  defense  industry’s  viewpoint  on  using  M&S  to  significantly  reduce  cost  over  the  entire 
acquisition  cycle  of  major  weapons  systems.  The  study  was  sponsored  by  the  Assistant 
Secretary  of  the  Navy  (RDA)  and  the  team  included  representatives  from  13  major  corporations 
and  six  government  or  university  labs. 

The  current  M&S  environment  is  characterized  as  using  specialized  tools  with  limited 
scope  and  scalability,  and  with  little  interoperability.  There  are  many  independent  databases 
with  ad  hoc  configuration  management  and  traceability.  Initial  process  modeling  tools  are  not 
widely  employed  so  that  the  many  advantages  associated  with  early  evaluation  are  not  realized. 

The  study’s  vision  for  future  M&S  employment  has  as  its  foundation  an  integrated  M&S 
environment  and  an  iterative  process.  All  stages  of  the  acquisition  process  interact  with  the 
common  product  data  repository.  This  permits  use  of  a  comprehensive  multi-domain  product 
model  definition,  which  extends  from  operational  need  to  downstream  O&S  activities  as  well  as 
system  modification  and  upgrade.  A  distributed  object-oriented  design  database  can  also  provide 
users  with  transparent  hypermedia  style  access  and  seamless  integration  of  all  critical 
engineering  disciplines.  An  iterative  spiraling  process  permits  rapid  evaluation  of  multiple 
options,  while  using  electronic  exchange  of  system  models  to  speed  up  information  flow.  This 
facilitates  integrated  process  teams  that  span  government  and  industry  organizations. 


The  technical  payoffs  or  advantages  of  using  this  approach  include:  quick  impact 
assessment  for  changes  in  requirements,  rapid  evaluation  of  multiple  design  options,  opportunity 
to  manage  technology  insertion,  more  efficient  understanding  and  communication  of  design  data, 
streamlined  integration  and  testing,  a  basis  for  reuse  and  re-engineering  of  existing  designs,  and 
better  government  and  industry  accountability.  The  team’s  draft  conclusions  were  that  this 
approach  can  dramatically  reduce  cost  and  schedule  while  improving  product  quality.  The  study 
goal  is  to  achieve  a  50%  or  5  year  (for  large  programs)  time  reduction  to  IOC  for  most  programs 
(depending  on  types  of  learning  curves). 

4.  DoD  Manufacturing  Initiatives 

The  DoD  has  a  number  of  initiatives  for  flexible,  computer-integrated  manufacturing 
(CIM).  The  Navy  Rapid  Acquisition  of  Manufactured  Parts  (RAMP)  Program  is  developing 
software  and  hardware  modules  for  improving  manufacturing  engineering  and  production 
management.  RAMP  modules  are  operational  in  12  Navy  and  Amiy  facilities.  The  Army 
Flexible  CIM  (FCIM)  program  (an  offshoot  of  RAMP)  focused  on  establishing  electronic  links 
between  inventory  management  and  design  engineering  sites  and  on  strengthening  the  analytical 
capability  of  these  sites. 

FCIM  involves  application  of  computers  and  related  technologies  to  rapidly  design, 
develop,  produce,  and  maintain  Army  products,  primarily  component  parts.  Program  goals  are  to 
provide  a  smooth  and  efficient  process  for  modifying  designs  and  producing  parts  in  both  low 
and  high  volumes  with  significantly  decreased  overall  acquisition  time,  lower  cost,  lower 
inventory  levels,  and  higher  quality. 

The  Army  Materiel  Command’s  (AMC)  Tobyhanna  Army  Depot  is  the  lead  on  this 
DARPA  funded  program.  AMC  works  closely  with  the  University  of  Iowa,  Lehigh  University, 
and  a  consortia  of  companies  on  FCIM,  as  well  as  assisting  with  implementation  of  RAMP. 

CAD/CAM  tools  and  environment  to  be  used  in  conjxmction  with  FCIM  are  to  be 
compatible  v\dth  PDES/STEP  (Product  Data  Model  using  STEP/Intemational  Standards  for  the 
Exchange  of  Product  Data)  and  with  DARPA’ s  Initiative  for  Concurrent  Engineering  (DICE) 
network.  M&S  programs  being  used  for  factory  automation  include  WITNESS  and  VERT.  The 
FCIM  goal  is  to  cut  existing  procurement  lead  time  from  500  to  30  days  by  improvements  in 
equipment,  communications,  people,  software,  and  business  practices. 

FCIM  is  a  network  of  modules  that  link  inventory  control,  configuration  management, 
manufacturing/contractor  sites.  The  overall  strategy  is  to  link  appropriate  enterprise  elements 
and  information  requirements  by  using  existing  and  developing  technology.  For  the  Electronics 
Module,  results  include  a  66%  reduction  in  cycle  time,  $3M  in  hard  dollar  cost  savings,  $3.8M  in 
cost  avoidance,  improved  communications  between  sites  and  customers,  and  improved  re¬ 
engineering  and  reverse  engineering  for  communication  end  items. 


The  FCIM  program’s  benefits  are  being  disseminated  to  other  AMC  sites  and  industry, 
including  use  of  innovative  concepts  such  as  teaching  factories,  contractor  implementation 
programs,  and  networks.  FCIM  seeks  joint  interaction  and  harmonization  with  other  programs, 
including: 

-  Joint  Computer-Aided  Acquisition  and  Logistic  Support  (JCALS), 

-  Joint  Engineering  Data  Management  Information  Control  System  (JEDMICS), 

-  Configuration  Management  Information  System  (CMIS), 

-  Automated  Document  Conversion  System  (ADCS),  and 

-  Product  Definition  Standard  System  (PDSS). 

5.  National  Information  Infrastructure  (Nil)  Manufacturing  Report 

Many  organizations  and  individuals  are  working  to  improve  the  NIL  An  improved 
infrastructure  will  provide  benefits  in  many  areas,  including  manufacturing.  A  recent  paper, 
“Manufacturing  and  the  Nil”,  provides  good  insights  on  trends,  benefits,  and  requirements  for 
future  improvement.  The  potential  benefits  of  Nil  include:  ability  to  efficiently  and  quickly 
move  data  within  and  between  organizations  allowing  collaboration  among  entities  at  distributed 
locations;  enabling  rapid  vertical  and  horizontal  integration  of  products  and  companies; 
accelerating  first  time  optimization  of  products  from  design  to  production;  and  permitting  virtual 
M&S,  testing,  and  manufacturing. 

One  study  found  that  advanced  manufacturing  techniques  that  enable  the  rapid 

exchange  of  information  not  only  increase  quality  and  cut  the  number  of  design  changes 

by  50%,  but  also  reduce  total  cost  by  30  to  60%,  development  time  by  35  to  60%,  design 

and  product  defects  by  30  to  80%,  and  scrap  work  by  58  to  75%. 

The  report  describes  many  innovations  and  improvements  that  are  being  employed  by 
other  major  corporations,  including  ways  to  improve  the  national  information  infrastructure. 
Many  of  the  initiatives  are  led  by  consortia.  Examples  of  key  consortia  and  initiatives  related  to 
manufacturing  include: 


-  Manufacturing  Extension  Partnership  (MEP)  sponsored  by  NIST  whose  major  efforts  include  LINKS  (an 
information  infrastructure  program)  and  TECNet  (a  LINKS  pilot  program  to  connect  extension  centers  and  users 
with  key  data  sources) 

-  National  Industrial  Information  Infrastructures  Protocols  (NIIIP)  Consortium  led  by  IBM,  is  partially 
sponsored  by  the  TRP,  which  is  developing  computer  protocols,  software  architecture,  tools,  etc.,  for  linking 
together  virtual  firms 

-  Rapid  Response  Manufacturing  (RRM)  Consortium  made  up  of  4  large  US  manufacturers,  GM,  Ford, 
Texas  Instruments,  and  United  Technologies,  with  the  intent  of  creating  tools  that  “better  assure  an  accurate  first 
part,  achieve  one-pass  product  designs  ...  and  provide  simultaneous  consideration  of  manufacturing  process 
constraints  in  the  generation  of  initial  designs” 

-  National  Initiative  for  Product  Data  Exchange  (NIPDE),  a  partnership  to  coordinate  the  development  of 
STEP  through  use  of  an  approved  standards  development  plan,  hosted  by  NIST/DoC 


-  PDES,  Inc.,  is  a  joint  industry/govemment  consortium  to  speed  the  development  and  implementation  of 
STEP  and  which  is  comprised  of  various  groups  and  programs  including  a  demonstration  project  for  Advanced 
Weapons  System  development 

-  Continuous  Acquisition  and  Life-Cycle  Support  (CALS)  is  a  joint  government-industry  initiative  with 
the  goal  of  allowing  all  parts  of  an  enterprise  to  operate  from  a  common  digital  database,  in  real  time,  for  all  life- 
cycle  phases 

-  Systems  Integration  for  Manufacturing  Applications  (SIMA)  is  a  NIST  program  to  develop  a  set  of 
integrated  manufacturing  systems  using  High  Performance  and  Communication  (HPCC)  technology  with  the  goal  of 
creating  an  Advanced  Manufacturing  System  and  Networking  Testbed  (AMSANT) 

-  Technologies  Enabling  Agile  Manufacturing  (TEAM)  project  by  DoE  helps  industry  collaborate  with 
and  use  DOEs  capabilities  in  key  manufacturing  thrust  areas 

-  Agile  Manufacturing  Initiative  is  a  DARPA  and  NSF  program  to  develop  the  prototype  information 
infrastructure  that  enables  ‘virtual  companies’  to  be  formed  for  design  and  manufacturing 


Significant  barriers  including  technical,  cultural,  and  financial,  exist  to  advanced 
manufacturing  through  use  of  an  NIL  Inexperience  with  information  technologies  and  the 
perceived  threat  of  new  systems  versus  traditional  ways  of  doing  business  is  a  barrier  that  must 
be  overcome  by  the  application  of  proper  training  and  education.  Uncertainty  and  risk  associated 
with  new  models  and  infrastructure  is  a  problem  to  be  addressed.  Selecting  and  configuring  new 
information  equipment  to  upgrade,  replace,  or  use  in  conjunction  with  old  equipment  offers 
technical  challenges.  The  report  provides  data  showing  that  rates  for  adopting  and  investing  in 
technology  is  a  traditional  problem  for  the  US  when  compared  to  economic  competitors.  This  is 
summarized  in  the  following  statements: 

...  most  of  the  370,000  small-  and  medium-sized  manufacturers  in  the  US,  who 
comprise  nearly  98%  of  all  manufacturing  firms,  have  neither  the  expertise,  time,  nor 
resources  to  modernize  their  manufacturing  processes  without  some  assistance. 

Moreover,  small  and  large  firms  alike  are  inhibited  by  the  difficulty  they  find  identifying 
and  understanding  technology  trends,  generating  adequate  investment  decision-making 
and  strategic  planning  models,  implementing  new  technologies  and  migrating  from  old 
legacy  systems. 

...the  result  of  these  trends  is  that,  for  the  most  part,  where  information 
technologies  have  been  developed  and  applied  to  manufacturing  operations,  it  has  been 
done  with  a  high  degree  of  sophistication  yet  with  a  narrow  focus  that  makes  integration 
of  these  manufacturing  technologies  not  economically  feasible.  As  a  result,  while  US 
manufacturers  excel  in  product  R&D  and  innovation  and  the  automation  of  individual 
components  of  the  overall  manufacturing  process,  they  frequently  fare  less  well  in  the 
combination  of  individual  technologies  into  an  integrated  manufacturing  system, 
embodying  such  concepts  as  concurrent  engineering,  total  quality  management,  and  just- 
in-time  inventory  control. 


6.  Center  of  Excellence  (COE)  for  Best  Manufacturing  Practices  (BMP) 

The  COE  for  BMP  provides  a  range  of  tools  to  assist  program  managers  in  all  aspects  of 
the  manufacturing  process,  from  design  through  production.  The  COE  for  BMP  is  sponsored  by 
DoN  and  Department  of  Commerce  and  is  available  to  assist  any  manufacturing  organization  that 
seeks  to  improve  its  practices.  Many  large  defense  contractors  are  participants  in  the 
organization  and  make  use  of  the  BMPs.  The  tools  include: 

-  KnowHow,  a  knowledge  database  of  acquisition  regulatory  guidance 

-  SpecRite,  a  knowledge  database  that  allows  specification  generation 

-  TRIMS,  a  tool  providing  insight  to  the  systems  engineering  process  as  well  as  risk 

assessment  and  review 

-  BMP  Database,  an  experience  database  of  best  practices  that  have  been  verified  and 

which  include  POCs  for  further  assistance 

-  BMPnet,  a  communications  system 

KnowHow  is  a  key  tool  providing  guidance  and  compliance  information  necessary  for 
acquisition  programs.  It  can  be  used  as  a  learning  tool  for  new  personnel,  and  it  can  serve  as  a 
search  engine  to  determine  specific  information.  TRIMS  and  SpecRite  are  management  tools. 
Several  success  stories  from  the  program  are  available. 

The  BMP  Database  documents  best  practices  for  issues  related  to  the  entire 
manufacturing  process.  The  best  practices  are  determined  through  visits  and  surveys  to 
manufacturing  sites  by  a  team  of  experts  in  the  necessary  disciplines  or  functions.  A  report  is 
published,  with  key  observations  divided  into  two  categories;  best  practices  and  information. 
Best  practices  are  those  methods  recognized  as  most  successful  and  competitive  within  an 
industry,  such  that  other  firms  may  want  to  use  them  as  benchmarks  and  emulate  them.  In 
general,  proprietary  data  is  not  included  in  the  report  or  the  database,  but  may  be  obtained 
through  the  POC. 

The  BMPnet  helps  to  disseminate  important  information  to  rapidly  reach  a  wide 
audience.  In  summary,  the  COE  for  BMP  helps  keep  US  manufacturing  companies  and  other 
firms  highly  competitive  through  sharing  of  the  best  manufacturing  ideas  and  methods. 

7.  Joint  Advanced  Distributed  Simulation  (JADS) 

The  goal  of  this  DoD  program  is  to  explore  and  increase  the  utility  of  advanced 
distributed  simulation  (ADS)  for  T&E  of  complex  military  systems  such  as  aircraft,  missiles,  etc. 
JADS  was  chartered  in  October  1994  by  OSD  to  investigate  the  use  of  ADS  for  both 
developmental  and  operational  T&E.  The  program  is  investigating  the  utility  of  ADS  including 
critical  constraints  and  concerns,  and  requirements  that  must  be  met  by  an  ADS  system  to 
support  a  more  complete  T&E  capability  in  the  future. 

JADS  testing  concentrates  on  the  performance  of  ADS  and  its  components,  not  on  any 
particular  system.  Data  gathering  and  conclusions  will  focus  on  network  performance. 


relationships  between  data  latencies,  and  ADS  induced  data  anomalies.  Testing  areas  have  been 
selected  to  allow  comparison  of  time,  cost,  complexity,  and  validity/credibility  of  data,  as  well  as 
test  activities  that  would  be  infeasible  without  use  of  ADS  technology. 

The  key  issue  pertaining  to  the  utility  of  ADS  for  T&E  is  determining  the  cost/benefits 
and  the  value  added.  Another  issue  pertains  to  critical  constraints  and  methodologies  for  using 
ADS.  JADS  will  research  concerns  such  as  simulation  identification  and  capability  evaluation, 
integration  of  varied  models,  interface  and  network  development,  support  infrastructure,  and 
verification  and  validation  of  networks.  A  third  issue  is  what  requirements  are  needed  for  ADS 
to  support  a  more  complete  T&E  capability  in  the  future  in  terms  of  network  and  simulation 
standards  for  data  structures,  security,  fidelity,  scalability,  emission  representations,  and  reactive 
terrain. 


The  program  currently  includes  two  approved  tests:  a  System  Integration  Test  and  an 
end-to-end  test.  These  tests  are  scheduled  for  completion  in  1998.  A  third  Electronic  Warfare 
test  is  also  being  considered.  These  tests  will  not  provide  sufficient  data  for  overall  conclusions 
about  the  utility  for  ADS  for  the  full  spectrum  of  T&E,  therefore  information  will  also  be  taken 
from  other  test  areas.  The  goal  is  to  obtain  hard  analytical  proof  for  difficult  issues  regarding 
ADS  in  testing. 

ADS  is  a  means  to  interface  live,  virtual,  and  constructive  players  so  they  can  interact 
with  each  other  on  a  common  playing  field.  Protocols  standardized  by  the  Institute  of  Electronic 
and  Electrical  Engineers  (IEEE)  have  helped  make  Distributed  Interactive  Simulation  (DIS)  a 
reality;  however  these  protocols  are  not  necessarily  sufficient  for  T&E  purposes.  ADS  uses 
similar  concepts  as  DIS,  but  goes  further  to  achieve  the  enhanced  capabilities  required  for  full 
fidelity  T&E.  ADS  will  be  more  conceptual  and  therefore  more  flexible.  ADS  may  use  the  High 
Level  Architecture  (HLA)  currently  under  development.  ADS  permits  a  central  computer  to 
control  the  simulation  nodes,  and  allows  a  central  server  to  maintain  the  environmental  data  base 
and  provide  it  to  other  nodes  on  demand.  Instead  of  use  of  Protocol  Data  Units  (PDUs)  required 
by  DIS,  ADS  can  use  a  more  flexible  implementation  that  sends  an  entity  only  that  information 
which  is  required. 

The  potential  for  ADS  to  enhance  T&E  is  significant.  ADS  can  help  in  test  planning  by 
helping  determine  which  field  tests  are  most  critical.  Testers  can  design  tests  to  address  mission- 
level  MOEs  using  direct  measurement  and  end-to-end  tests.  Simulation  can  provide 
environments  that  are  not  possible  to  test  in  the  field,  and  allows  for  complete  post-test 
extrapolation  to  differing  scenarios  and  test  environments.  Some  specific  issues  and  risks  are: 


•  complexity  in  terms  of  scheduling  networks,  test  assets,  operators,  etc. 

•  latency  associated  with  geographic  distribution  of  test  assets 

•  interfacing  and  reacting  problems  between  simulated  and  live  players 

•  similar  interface  problems  between  terrain  database  and  actual  terrain 

•  added  complexity  in  the  VV&A  of  a  network  of  models,  versus  single  ones 

•  coordination  problems  with  numerous  participating  organizations 

•  data  management  for  distributed  testing  in  terms  of  range  ‘truth’  when  using  dissimilar  ranges 

•  technological  maturity  of  ADS,  especially  in  terms  of  stringent  realism  requirements 


•  reliability  associated  with  an  infrastructure  for  ADS 

•  whether  cost  for  distributed  testing  will  be  less  than  for  traditional  testing 


8.  TARDEC’s  Tracked  Vehicle  Work  Station  (TVWS) 

TARDEC  employs  the  virtual  prototyping  process  to  conduct  early  exploration  and 
evaluation  of  new  vehicle  concepts  without  actually  building  a  physical  vehicle.  The  process 
lends  itself  to  continuous  user  participation,  including  soldiers  who  will  ultimately  use  the 
system  in  the  field,  as  well  as  the  developers  (government-contractor  team).  Advanced  computer 
simulation  enables  early  evaluation  of  new  concepts  without  actually  building  a  prototype 
vehicle. 

TARDEC  employs  the  TVWS  to  facilitate  concept  exploration  and  evaluation.  Engineers 
can  select  and  assemble  a  concept  tracked  vehicle  using  appropriate  parts  from  a  CAD  parts 
database.  They  then  select  appropriate  test  environments  and  conduct  virtual  testing.  By 
reviewing  test  results  using  various  techniques,  the  engineer  can  validate,  refine,  or  discard  the 
concept,  quickly  and  inexpensively.  The  TVWS  has  available  the  necessary  component  parts, 
scenarios,  test  plans,  and  other  related  databases,  as  well  as  the  supercomputing,  animation, 
display,  and  other  capabilities  needed  to  evaluate  the  tracked  vehicle  concept. 

The  TVWS  also  facilitates  progression  to  prototyping  and  design  activities  and  to 
development  and  use  of  crew  stations  and  motion  simulators.  TVWS  capabilities  can  flesh  out 
the  3-D  virtual  prototype  so  that  the  user  can  explore  inside  the  vehicle  and  obtain  valuable 
human-machine  interface  feedback.  By  interfacing  with  other  constructive  and  virtual 
simulations,  the  user  is  able  to  ‘fight’  the  vehicle  and  better  understand  and  improve  its  design 
before  hardware  fabrication.  Later,  the  design  evolves  into  a  crew  station  envelope  or  simulator, 
which  at  first  is  not  realistic  in  terms  of  configuration,  but  which  is  realistic  in  terms  of  human- 
machine  interfaces.  The  envelope  can  be  used  imder  static  conditions,  or  dynamic  conditions 
using  motion-based  simulators.  The  crew  station  envelope  is  connected  via  the  Distributed 
Simulation  Internet  to  enable  users  to  evaluate  the  effect  of  the  conceptual  vehicle  on  tactics  and 
force  effectiveness. 

9.  Automotive  Consortia 

Closely  associated  with  TARDEC  and  TACOM  are  a  series  of  organizations  of  which  the 
most  key  are  the  National  Automotive  Center  (NAC),  the  Automotive  Research  Center  (ARC), 
and  the  Industry/University  Cooperative  Research  Center  (I/UCRC)  for  Simulation  and  Design 
Optimization  of  Mechanical  Systems.  Each  of  these  is  partially  fimded  by  DoD,  and  attempts  to 
further  the  state-of-the-art  in  terms  of  optimizing  design  and  manufacture  of  vehicle  systems. 

The  NAC’s  role  is  to  forge  closer  integration  between  legacy  systems  and  evolving 
infrastructure  improvements.  TARDEC  seeks  to  leverage  commercial  developments  and  apply 
them  to  Government  needs.  The  NAC  is  especially  involved  in  helping  define  a  standard 
technical  architecture  to  achieve  a  more  integrated  M&S  approach. 


The  ARC  has  the  role  of  conducting  R&D  for  improved  automotive  manufacturing, 
especially  for  flexible,  agile  modeling  systems.  They  work  with  many  types  of  M&S  tools  with 
the  goal  of  achieving  better  tool  performance  and  interoperability.  Their  five  research  areas  are: 
vehicle  terrain  dynamics,  vehicle  hardware/human  interface  simulation,  M&S  of  vehicle 
structures,  advanced  propulsion  simulation,  and  system  integration.  The  ARC  uses  an 
organizational  structure  called  the  Quad  Concept  to  ensure  that  every  project  is  represented  by 
university  students,  university  faculty,  industry  researchers,  and  Government  scientists.  This 
ensures  that  technology  that  is  developed  is  in  tune  with  industry  and  Government  needs,  and  is 
available  for  practical  implementation. 

The  I/UCRC  group  is  located  at  the  University  of  Iowa  and  has  as  its  objective 
development  and  distribution  of  advanced  analytical  simulations.  Sponsors  include  TACOM, 
NASA,  and  the  National  Science  Foundation  (NSF).  A  key  tenet  of  the  Center’s  research  is  that 
there  is  a  need  for  an  environment  in  which  advanced  CAD/C  AM/CAE  tools  can  be  concurrently 
brought  to  bear  in  support  of  engineering  developments  that  involve  a  broad  range  of  disciplines. 
The  Center  helped  develop  the  TVWS.  Other  basic  and  on-going  technical  objectives  for  the 
Center  are:  dynamic  simulation,  operator-in-the-loop  simulation,  dynamic  stress  and  lifetime 
prediction,  structural  design  sensitivity  analysis  and  optimization,  pilot  and  driver  projects  for 
operator-machine  interactive  simulations,  driving  simulators,  and  SBD  for  military  system 
supportability  and  human  factors. 

10.  NAVSEA’s  Smart  Product  Model  (SPM) 

The  NAVSEA  SPM  permits  exploration  of  various  concepts  or  designs  before  beginning 
to  downselect  the  best.  This  program  relates  to  the  DARPA  simulation  based  design  (SBD) 
concept.  Most  virtual  activities  are  in  the  early  stages  of  the  acquisition  process  (i.e.,  concept 
exploration,  prototyping  and  design);  however,  they  clearly  link  up  with  all  stages  of  the  life 
cycle.  The  SPM  encompasses  all  stages  of  the  life  cycle,  including  lesson  learned  from 
operational  systems.  The  Navy  process  relies  heavily  on  the  IPPD  team  and  on  multiple 
iterations  of  virtual  design,  build,  and  test,  prior  to  the  actual  bending  of  metal.  The  design  and 
analysis  are  based  on  assessing  a  series  of  ship  functionalities  such  as  signature,  vulnerability, 
seakeeping,  and  so  forth. 

Data  fi'om  the  various  stages  of  the  model  and  for  the  various  functionalities  are  stored  in 
a  common  repository  (the  Smart  Product  Model)  for  access  by  the  range  of  system  developers 
and  other  users.  The  SPM  interacts  with  the  virtual  environment  and  ‘supporting’  functions 
including  manufacturing,  logistics,  financial,  program  management,  and  operational  assessment. 

1 1 .  Lockheed  Martin’ s  Virtual  Development  Environment 

Lockheed  Martin  is  using  the  Air  Vehicle  Virtual  Development  Environment  to  fully 
exploit  the  value  of  virtual  development.  Industrial  sectors  moving  most  swiftly  toward  virtual 
prototyping  (VP)  (aircraft,  heavy  machinery,  and  automotive)  are  characterized  by  high 
investment  costs  for  items  such  as  tooling,  need  for  prototype  development,  desire  for  reduced 


cycle  time  in  order  to  rapidly  integrated  technology  and  other  improvements,  and  high  product 
support  and  O&S  cost.  Their  move  toward  increased  use  of  M&S  is  fueled  by  expected 
improvements  of  30  to  50%  reduction  in  design  cycle  time  and  cost,  25  to  50%  reduction  in 
assembly  time  and  cost,  up  to  25%  reduction  in  tooling  cost,  50%  reduction  in  time-to-market, 
and  other  significant  reduction  in  product  support  cost.  Table  IV-1  provides  some  types  and 
indicators  of  savings  and  increased  effectiveness. 

Lockheed  Martin  uses  M&S  in  conjunction  with  digital  product  and  process  definition. 
VP  provides  a  vehicle  to  test  concepts  and  designs  prior  to  committing  resources,  evaluate  new 
concepts,  verify  design  integrity,  and  in  many  instances,  eliminate  the  need  for  hardware 
mockups.  VP  is  used  for  all  stages,  from  design  to  production  and  support.  The  major  elements 
of  VP  are  CAD  defined  models  and  static  or  ‘flythrough’  evaluation  models.  The  latter  are  used 
for  detecting  interference,  mismatch,  and  tolerance  problems,  and  to  help  with  system  support 
assessments  and  performance  and  analytical  assessments  in  ‘real’  environments. 

Lockheed  Martin  has  developed  an  M&S  process  vision  that  includes  product, 
manufacturing,  and  factory/enterprise  fimctions.  The  process  spans  functions  from  requirements 
definition  to  fleet  operations,  and  helps  integrate  product  configuration/design,  manufacturing 
concepts,  tooling,  factory  flow,  and  factory/enterprise  design.  Frequent  looping  through  the 
operations  analysis  step  ensures  that  all  aspects  of  the  design  are  updated  and  continue  to  best 
meet  the  established  requirements.  A  hierarchy  of  simulation  capabilities  are  needed  to  support 
the  development  process.  Levels  of  capability  include  integrated  CAD  systems,  rule-based 
design  support,  accurate  3-D  solid  models/visualizations,  and  physics-based  simulation.  Key 
virtual  development  environment  capabilities  are  used  for  both  concept  development  and  detailed 
development  using  many  common  tools  and  a  common  design  database. 

Physics-based  simulations  provide  a  first  approximation  of  how  the  product  will  perform 
and  how  the  processes  (in  terms  of  production,  logistics  and  O&S  support,  etc.)  will  rate  in  terms 
of  effectiveness.  First,  however,  solid  design  capabilities  are  used  to  construct  the  virtual  air 
system  model  in  terms  of  systems  engineering  and  meeting  the  integrated  needs  and 
requirements  of  the  system.  The  rule  based  conceptual  designs  permit  configuring  the  system 
and  its  subsystems;  developing  process  and  resource  information  relating  to  manufacturing  and 
tooling;  and  providing  preliminary  cost  assessments  (increasingly  important  in  the  new 
acquisition  environment)  for  all  aspects  of  the  program,  including  all  areas  relating  to  production, 
test,  and  deployment/operations.  M&S  also  allows  earlier  and  better  visualization  of  the  system 
as  it  will  evolve.  This  is  useful  for  management  as  it  evaluates  not  only  the  product  but  also  the 
tooling  and  factory  requirements  and  the  maintenance  and  support  needs. 


TABLE  IV-1 :  IMPROVEMENT  INDICATORS  BASED  ON  USE  OF  M&S 

Process  M&S  Potential  Affordability 

Activity _ Capability _ Cost  Savings _ Impact 


Design 

optimization 

Use  rule-based  models  to  select  low- 
cost  Fab.  Processes,  and  standard 
parts,  toolings,  and  processes 

•  Reduced  part  and  assy,  cost 

•  Reduced  tool  design  cost 

•  Reduced  tooling  rqmts. 

Development 

Development 

Production 

Part  Design 

•  Solid  modeling 

•  Computer  simulated  assembly  with 
3-D  interactive  graphics 

•  Eliminate  physical  mock-ups  at 
$30M  each 

•  30-50%  cost  reduction  in 
design/verification  steps 

•  20-25%  reduction  in  first  article 
(and  on)  assy,  cost 

Development 

Development 

Production 

Loads  analysis 
for  tooling 

Finite  element  analysis  coupled  with 
manufacturing  process  simulations 

Eliminate  scrap  and  rework  due  to 
tool  loadings  and  tolerance  changes 

Development 
&  Production 

Conceptual 

design 

optimization 

IPD  and  analysis  models  for  IR  and 

RF  signature,  RF  emission, 
aerodynamics,  structure 

•  Reduction  in  design  and  analysis 
spans,  cost,  and  rework 

•  Reduction  in  number  of  physical 
tests/better  test  planning 

Development 

Development 

RM&S  design 

•  Model  and  simulate  support 

•  Processes  and  designs  for 
supportability  evaluation: 

-  Ergonomic 

-  Removal/replacement  paths 

•  Potential  reduction  in  maintenance 
man-hours  up  to  a  factor  of  3 

•  Eliminate  mock-ups 

Support 

Development 

Loads  analysis 

•  Use  integrated  CFD  and  structural 
analysis  models  to  determine  initial 
pressure  loads 

•  Use  integrated  structural  analysis 
and  Navy-unique  loads  criteria 

•  Reduced  design  span 

•  Reduced  wind  tunnel  testing 

Development 

Development 

Weapons 
carriage  and 
separation 
certification 

Use  integrated  CFD  loads  analysis 
(virtual  wind  tunnel)  to  verify  carriage 
and  separation 

•  Wind  tunnel  test  time  reduced  or 
eliminated 

•  Elimination  of  most  carriage  and 
separation  of  flight  tests 

Development 

Development 

Capacity 

planning 

Simulate  and  evaluate  processes  and 
costs  for  varying  production  rates 

Achieve  rate  insensitive  costs 

Production 

Production 

training 

3-D  visual  simulation  of 
manufacturing  process 

Flattened  learning  curves 

Production 

Manufacturing 
design  and 
planning 

Simulate  NC  programs  to  verify 
paths,  interactions,  interferences,  in¬ 
feasible  positions,  mismatches,  etc. 

•  Eliminate  rework 

•  Reduced  development  effort 

Production 

Development 

Tolerance  and 

variability 

assessment 

M&S  design  and  tooling  integration 
with  tolerance  build-ups  (use  SPC 
data  with  assembly  simulations) 

Assembly  cost  reduction 

Production 

Manufacturing 
design  and 
planning 

Model  and  simulate  manufacturing 
processes  and  factor  flow  (production 
management)  and  enterprise 
operations 

•  Reduced  inventory/WIP 

•  Reduced  factory  span 

•  Learning  curve  lowered,  lower  TI 
cost,  greater  efficiency 

Production 

Production 
planning  and 
training 

Ergonomic  simulations  of 
manufacturing  processes 

•  Improved  salary/lower  overhead 
costs 

•  Reduced  crew  loading  rqmts. 

Production 

Manufacturing 

planning 

Simulate  part/tool  mechanical  and 
thermal  interactions 

•  Reduced  tooling  and  capital  rqmts. 

•  Elimination  of  scrap  and  rework 

Production 

V.  FINDINGS  AND  RECOMMENDATIONS 


This  study  set  out  to  assess  M&S  effectiveness  in  the  weapon  systems  acquisition 
process.  The  resounding  results  were  that  M&S  was  effective  in  reducing  risk  throughout  the 
development  cycle,  improving  system  performance,  and  lowering  total  life  cycle  costs.  The 
documented  metrics  revealed  this.  Most  programs  cited  cost  avoidance  or  resource 
conservation.  Specific  CAD/CAM/CAE  tools  provide  synthetic  representations  of  system 
components,  the  future  system,  and  the  production  process.  The  integration  of  these  tools  and 
their  integrated  system  database  provide  the  IPTs  the  continuity  to  make  optimal  decisions 
while  concurrently  evaluating  development.  The  benefits  of  using  M&S  parallel  those  of  the 
IPPD  itself.  The  improved  decision  making  ability,  along  with  optimal  design,  assembly, 
employment,  and  user  involvement  are  a  great  improvement  over  the  ^^traditional”  acquisition 
process. 

There  is  a  recognized  need for  technology  to  be  used  by  the  acquisition  community  as  it 
re-engineers  itself  into  using  a  team-based  approach.  This  message  has  been  given  by 
Secretary  of  Defense,  Dr.  Perry,  Under  Secretary  of  Defense  for  Acquisition  &  Technology, 

Dr.  Kaminski,  and  others  as  the  benefits  reaped from  shifting  from  a  stovepipe  process  to 
Simulation  Based  Acquisition  become  more  obvious. 

Modeling  and  simulation  tools  and  processes  are  being  used  efficiently  and  effectively 
in  each  of  the  Services.  Examples  of  early  and  continuous  interaction  between  users, 
developers,  and  testers  are  evident  in  several  programs.  The  use  of  M&S  and  virtual 
environments  to  quickly  examine  the  impact  and  results  of  decisions  have  formed  the  basis  for 
better  decision  making.  Risk  is  better  managed  through  analysis  of  virtual  mockups  in  virtual 
environments.  There  are  many  examples  of  reduction  in  cycle  time  and  greater  flexibility  in 
decision  making  through  the  use  of  M&S. 

There  are  several  challenges  that  the  study  team  identified  that  need  resolution.  The 
issue  of  proprietary  data  and  models  is  a  contentious  one  with  many  implications.  This  issue 
was  raised  by  both  Government  and  contractor  personnel,  at  nearly  every  site,  as  a  point  for 
discussion.  Some  of  the  other  identified  challenges,  such  as  security  of  data  and  availability  of 
data  descriptions,  may  be  resolved  as  the  acquisition  community  grows  to  more  fully  embrace 
the  distributed  environment. 

The  technical  challenges  identified  are  not  beyond  the  capability  of  improving 
technology  to  resolve,  although  prudent  investments  or  proper  encouragement  might  well 
speed  and  enhance  the  usefulness  of  the  technical  solutions.  Cultural  and  managerial  issues 
are  more  difficult  to  overcome.  They  are  often  the  by-product  of  current  initiatives  in  the 
acquisition  process.  Simulation  Based  Acquisition  will focus  on  alleviating  some  of  these 


concerns. 


The  study  team  noted  the  following  challenges: 


Type 

Challenges 

Technical 

0  Interoperability  of  M&S  Tools 

0  Availability  of  Data  Descriptions 

0  Security/Sensitivity  of  Data 
o  Physics-based  M&S 
o  Hardware  and  Software  Limitations 
o  Variable  Resolution 

Cultural 

0  Acquisition  Processes 

0  Incentives  for  M&S  Use 

0  M&S  Workforce 

0  Acceptance  of  M&S 

Managerial 

0  OSD  and  Service  Guidance 

0  Ownership  of  Data 

0  W&A  Requirements 

0  Funding  Process 

0  Use  of  System  Models 

One  challenge  is  how  to  institutionalize  M&S  into  the  acquisition  process  so 
that  it  is  used  productively  in  an  integrated  manner  in  weapon  system  acquisition.  The  culture 
is  ready  to  accept  the  changes  in  the  acquisition  process.  This  is  evidenced  by  the  programs 
using  M&S  piecemeal  throughout  the  Services.  The  difficulty  lies  in  encouraging  the 
program  managers  to  use  these  tools  more  efficiently  and  plan  use  of  M&S  seamlessly 
throughout  the  life  cycle. 

The  words  are  in  place  in  DoD  acquisition  documents  to  support  implementation  of 
Simulation  Based  Acquisition,  though  there  are  some  growing  pains  associated  with 
implementation  of  this  shift.  For  example,  the  Multiple  Launch  Rocket  System  program 
office  submitted  an  Extended  Range  -  MLRS  Test  and  Evaluation  Master  Plan  (TEMP), 
approved  through  the  Army  Staff,  that  was  heavily  supported  by  the  results  of  simulation.  The 
Office  of  the  Director,  Operational  Test  and  Evaluation  rejected  the  TEMP  and  directed  that 
36  more  live  rockets  be  fired  than  in  the  submitted  plan.  The  issue  is,  to  what  degree  can 
M&S  replace  or  augment field  tests?  There  is  no  universal  answer,  but  the  message  received 
in  part  by  the  community  is  that  the  thought  and  guidance  are  there,  but  the  implementation 
and  acceptance  are  not 

Recommendation:  An  immediate  action  is  to  institutionalize  the  use  of  models  and 
simulations  and  insure  that  the  community  is  knowledgeable  about  the  tools  available.  The 
Services  and  OSD  need  to  provide  more  responsive  guidance  relative  to  the  advent  of  better 
and  more  useful  simulation  tools.  Dialogue  is  needed  within  the  Services  and  between  the 
Services  and  OSD  to  effect  policy  on  standardization.  Program  managers  must  overcome  the 
management  and  cultural  challenges  that  present  barriers  to  the  effective  use  of  available 
technology. 


The  study  found  that  program  managers  and  their  staffs  are  not  well  informed  on 
M&S  tools  and  their  use  in  acquisition.  Many  program  managers  have  had  very  limited 
exposure  to  models  and  simulations  and  would  benefit from  a  short  block  of  instruction  which 
would  emphasize  the  successes  other  programs  have  experienced  by  using  M&S.  Defense 
Acquisition  University  has  begun  to  alter  the  curriculum  in  order  to  make  the  students  more 
aware  of  the  impact  of  using  current  technology  early  in  the  acquisition  process  and 
throughout  the  life  cycle  of  the  program. 

Recommendation:  Action  is  needed  to  provide  focused  information  on  the  availability 
and  capabilities,  to  include  success  stories,  of  M&S  to  weapon  system  acquisition  managers. 

A  frequently  discussed  topic  in  the  study  team's  visits  was  the  need  for  investment  early 
in  the  program  for  the  tools  that  would  be  useful  throughout  the  life  cycle.  For  programs 
such  as  the  F-22  and  the  Comanche,  there  was  no  choice  but  to  plan  to  invest  early  in  M&S 
and  plan  their  program  around  those  investments.  For  many  smaller  programs  the  decision  is 
more  difficult.  Guidance  is  very  general  and  there  is  little  incentive  for  program  managers  to 
commit  their  early  program  funds  to  technologies  for  which  they  don  V  see  an  immediate 
return.  This  is  complicated  by  the  fact  that  there  are  no  funding  lines  specifically  designated 
for  investing  in  the  simulations  necessary  to  support  the  acquisition. 

Investment  in  simulation  facilities  extends  beyond  the  money  invested  by  weapon 
system  programs.  The  anechoic  chambers  at  Edwards  Air  Force  Base  and  Patuxent  River 
Naval  Air  Station  and  the  wind  tunnel facility  and  computational  fluid  dynamics  capabilities 
at  Arnold  Engineering  Development  Center  are  examples  of  these  facilities.  The  Simulation 
Test  Acceptance  Facility  (STAF)  at  Redstone  Technical  Test  Center  was  funded  by  program 
managers,  but  is  a  facility  that  could  be  used  with  minor  modification  by  other  missile 
programs. 

Recommendation:  To  meet  the  challenge  of  institutionalizing  the  use  of  available 
technology,  the  Services  must  be  committed  to  providing  funds  for  modeling  and  simulation  at 
the  inception  of  the  program.  OSD  and  the  Services  should  commit  Science  and  Technology 
dollars  to  upgrade  capabilities  and  facilities  that  could  serve  many  weapon  system 
acquisitions.  Program  managers  should  be  encouraged  to  use  these  facilities  and  capabilities 
instead  of  contracting  to  have  their  own  system  specific  facilities  and  tools  built. 

There  is  currently  no  vehicle  to  get  information  on  M&S  capabilities  and  facilities  to 
the  programs  that  have  the  potential  to  utilize  the  assets.  It  would  be  useful  to  the  program 
managers  and  to  the  Research  and  Development  and  Test  and  Evaluation  activities  to  have  a 
source  for  investigating  M&S  capabilities  available  within  the  Defense  Acquisition 
community. 

Recommendation:  Develop  an  information  source  such  as  an  Internet  web  page  which 
would  list  capabilities  in  design;  tools  available,  programs  that  have  used  them,  activities  they 
can  contact  for  further  information,  etc.  The  same  capabilities  could  be  listed for  testing;  e.g., 
who  does  computational fluid  dynamics,  which  facilities  have  wind  tunnels  and  anechoic 


chambers,  have  any  programs  proven  out  this  technology  with  their  systems?  The  webpage 
could  also  be  used  to  identify  innovative  approaches  in  manufacturing  and  note  those  using 
virtual  manufacturing  environments. 

Industry  and  Government  are  cooperating  closely  in  many  areas  pertaining  to  M&S 
and  streamlined  acquisition.  In  reaction  to  global  or  national  competition  and  other 
pressures,  many  firms  have  modified  their  approach  to  product  development,  manufacturing, 
and  support.  Many  of  these  firms,  using  M&S  tools  and  processes  extensively,  have 
influenced  the  revolution  in  DoD  acquisition,  especially  firms  that  are  DoD  contractors,  and 
they  are  forging  closer  alliances  through  teamwork  and  IPT/IPPD  initiatives. 

One  key  effort  is  DARPA ’s  Simulation  Based  Design  (SBD)  program.  The  goal  of  the 
project  is  to  revolutionize  the  Acquisition  Process  for  complex  military  and  commercial 
products”  using  distributed,  collaborative  virtual  development  environments.  Another 
significant  partnership  is  the  collaboration  of  Industry,  Academia,  and  the  Government  in  the 
National  Automotive  Center  in  Detroit 

Recommendation:  Opportunities  to  cooperate  with  Industry  such  as  the  DARPA 
Simulation  Based  Design  programs  should  be  encouraged  and  continued.  There  appears  to 
be  great  potential  in  partnerships  such  as  the  National  Automotive  Center  where  both  the 
Government  and  Industry  benefit  from  investigating  new  technology.  There  should  be 
incentives  to  pursue  business  relationships  such  as  these  in  order  to  utilize  developing 
technology  more  efficiently. 

In  summary:  There  is  consistent  evidence  of  M&S  being  used  effectively  throughout 
the  acquisition  process  but  not  in  an  integrated  manner  across  programs  or  functions  within 
the  acquisition  process.  This  evidence  has  been  collected  from  individual  success  stories, 
though  the  benefits  are  not  readily  quantifiable  into  a  general  standard.  The  key  is  in 
focusing  on  the  integration  of  M&S  applications,  across  acquisition  programs  and  throughout 
the  process,  not  in  exploring  the  applications  themselves.  In  the  final  analysis,  M&S  will 
continue  to  grow  in  usage,  capability  and  total  contribution  to  the  acquisition  process.  The 
opportunity  exists  now  for  the  DoD  to  effect  Simulation  Based  Acquisition  by  focusing  on 
providing  the  opportunities  to  facilitate  the  change. 


APPENDIX  A  -  GLOSSARY 


3-D  Three  dimensional 

A&T  Acquisition  and  Technology 

AAAV  Advanced  Amphibious  Assault  Vehicle 

ACAT  Acquisition  Category 

ACETEF  Air  Combat  Environment  Test  and  Evaluation  Facility 
ADCS  Automated  Document  Conversion  System 

ADPA  American  Defense  Preparedness  Association 

ADS  Advanced  Distributed  Simulation 

AEDC  Arnold  Engineering  Development  Center  (USAF  RD&E  Center) 

AFATDS  Advanced  Field  Artillery  Tactical  Data  System 
AFB  Air  Force  Base 

AFFTC  Air  Force  Flight  Test  Center 

AHPCRC  Army  High  Performance  Computing  Research  Center 

ALSP  Aggregate  Level  Simulation  Protocol 

AMS  Affordable  Multi-Missile  Manufacturing 

AMC  Army  Materiel  Command 

AMRAAM  Advanced  Medium  Range  Air-to-Air  Missile 

AMSAA  Army  Materiel  Systems  Analysis  Activity 

AMSO  Army  Modeling  and  Simulation  Office 

ANSI  American  National  Standards  Institute 

ARC  Automotive  Research  Center 

ARL  Army  Research  Lab 

ASHPC  Advanced  Simulation  and  High  Performance  Computing 
ASME  American  Society  of  Mechanical  Engineers 

ASN  Assistant  Secretary  of  the  Navy 

ASNE  American  Society  of  Naval  Engineers 

ASW  Anti-Submarine  Warfare 

ATFMS  Acquisition  Task  Force  on  Modeling  and  Simulation 

B  Billion 

BFV  Bradley  Fighting  Vehicle 

BMP  Best  Manufacturing  Practices 

C2  Command  and  Control 

C3I  Command,  Control,  Communications  and  Intelligence 

C4  Command,  Control,  Communications  and  Computers 

CAD  Computer-Aided  Design 

CAE  Computer-Aided  Engineering 

CAIV  Cost  as  an  Independent  Variable 


CALS  Computer  Aided  Acquisition  and  Logistics  Support 

CAM  Computer-Aided  Manufacturing 

CASE  Computer  Aided  Software  Engineering 

CATIA  Computer  Aided  Three-Dimensional  Interactive  Application 

CATT  Combined  Arms  Tactical  Trainer 

CCTT  Close  Combat  Tactical  Trainer 

CE  Concurrent  Engineering 

CFD  Computational  Fluid  Dynamics 

CIM  Computer  Integrated  Manufacturing 

CinC  Commander  in  Chief 

CITIS  Contractor  Integrated  Technical  Information  Service 

CMIS  Configuration  Management  Information  System 

COE  Center  of  Excellence 

CORE  A  Common  Object  Request  Broker  Architecture 

COTS  Commercial  Off-the-Shelf 

CRDA  Cooperative  Research  and  Development  Agreement 

DA  Department  of  the  Army 

DARPA  Defense  Advance  Research  Projects  Agency 

DD,  MSEE  Deputy  Director,  Models,  Simulations,  and  Software  Evaluation 

DDR&E  Director,  Defense  Research  and  Engineering 

DEMVAL  Demonstration  and  Validation  (phase) 

DICE  DARPA  Initiative  for  Concurrent  Engineering 

DIS  Distributed  Interactive  Simulation 

DISA  Defense  Information  Systems  Agency 

DMSO  Defense  Modeling  and  Simulation  Office 

DoC  Department  of  Commerce 

DoE  Department  of  Energy 

DoD  Department  of  Defense 

DoN  Department  of  the  Navy 

DOT&E  Director,  Operational  Test  and  Evaluation 

DPA  Digital  Pre-Assembly 

DSB  Defense  Science  Board 

DSI  Defense  Simulation  Internet 

DSMC  Defense  Systems  Management  College 

DT&E  Developmental  Test  and  Evaluation 

DTSE&E  Director,  Test,  Systems  Engineering,  and  Evaluation 

EB  Electronic  Battlefield 

ECP  Engineering  Change  Proposal 

EDS  Engineering  Development  Simulator 

EMD  Engineering  and  Manufacturing  Development 

EOA  Early  Operational  Assessment 

ETMO  Education,  Training,  and  Military  Operations 

EW  Electronic  Warfare 


FDTE 

Force  Development  Test  and  Evaluation 

FCIM 

Flexible  Computer  Integrated  Manufacturing  (Army  program) 

FIS 

Firing  Impulse  Simulator  (Army  asset,  APG,  MD) 

FY 

Fiscal  Year 

GM 

General  Motors 

GOTS 

Government  Off-the-Shelf 

GPS 

Global  Positioning  System 

GWEF 

Guided  Weapons  Evaluation  Facility  (facility  at  Eglin  AFB) 

HLA 

High  Level  Architecture 

HMMWV 

High  Mobility  Multipurpose  Wheeled  Vehicle 

HPCC 

High  Performance  Computing  and  Communications  (program) 

HQ 

Headquarters 

HTI 

Horizontal  Technology  Integration 

HWIL 

Hardware-in-the-Loop 

IDA 

Institute  for  Defense  Analysis 

IEEE 

Institute  of  Electronic  and  Electrical  Engineers 

IGES 

Initial  Graphics  Exchange  Specification 

IOC 

Initial  Operational  Capability 

IPL 

Integrated  Priority  List 

IPPD 

Integrated  Product  and  Process  Development 

IPX 

Integrated  Product  Team 

IR 

Infrared 

IR&D 

Independent  Research  and  Development 

ISO 

International  Standards  Organization 

IT 

Information  Technologies 

IT&E 

Integrated  Test  and  Evaluation 

I/UCRC 

Industry/University  Cooperative  Research  Center 

JADS 

Joint  Advanced  Distributed  Simulation 

JAST 

Joint  Advanced  Strike  Technology 

JCALS 

Joint  Computer-Aided  Acquisition  and  Logistics  Support 

JEDMICS 

Joint  Engineering  Data  Management  Information  Control  System 

JITC 

Joint  Interoperability  Test  Center 

JMASS 

Joint  Modeling  and  Simulation  System 

JPO 

Joint  Program  Office 

JSF 

Joint  Strike  Fighter 

JSIMS 

Joint  Simulation  System 

JSOW 

Joint  Stand-Off  Weapon 

JSTARS 

Joint  Surveillance  Target  Attack  Radar  System 

JSTEB 

Joint  Synthetic  T&E  Battlespace 

KEP 


Kinetic  Energy  Penetrator 


LCC  Life  Cycle  Cost 

LCT  Longbow  Crew  Trainer 

LPD-17  Amphibious  Transport  Ship 

M  Million 

M&S  Modeling  and  Simulation 

MADEManufacturing  and  Design  Engineering  (DARPA  program) 

ManTech  Manufacturing  Science  and  Technology 

MAVWEST  Multiplex  Armaments,  Visionics,  Weapons  and  Electrical  Systems  Trainer 
MEP  Mission  Equipment  Package 

MICOM  Missile  Command  (Army) 

MLRS  Multiple  Launch  Rocket  System 

MOE  Measure  of  Effectiveness 

MOP  Measure  of  Performance 

MOO  Measure  of  Outcome 

MOU  Memorandum  of  Understanding 

MS&AModeling,  Simulation  and  Analysis  (USAF  program) 

MSBTF  Modeling  and  Simulation  Benefits  Task  Force 

MSRR  Modeling  and  Simulation  Resource  Repository 

MTIAC  Manufacturing  Technology  Information  Analysis  Center 

MTS  Moving  Target  Simulator  (TECOM  at  APG) 

NAC  National  Automotive  Center 

NASA  National  Aeronautics  and  Space  Administration 

NAVSEA  Naval  Sea  System  Command 

NAWCAD  Naval  Air  Warfare  Center  Aircraft  Division 

NAWCWPNS  Naval  Air  Warfare  Center  Weapons  Division 

NBC  Nuclear,  Biological,  and  Chemical 

NC  Numerical  Control 

NCSA  National  Center  for  Supercomputing  Applications 

NIDDESC  Navy/Industry  Digital  Data  Exchange  Standards  Committee 

Nil  National  Information  Infrastructure 

NIST  National  Institute  for  Standards  and  Technology 

NLOS  Non  Line  of  Sight 

NSF  National  Science  Foundation 

NSS  Naval  Simulation  System 

NSSN  New  Attack  Submarine  (also  NAS)  (Navy  Program) 

O&S  Operations  and  Support 

OASA  Office  of  the  Assistant  Secretary  of  the  Army 

OOT  Object  Oriented  Technologies 

OOTW  Operations  Other  Than  War 

ORD  Operational  Requirements  Document 


OSD  Office  of  the  Secretary  of  Defense 

OT  Operational  Test 

OTA  Operational  Test  Agency 

OT&E  Operational  Test  and  Evaluation 

PDES  Product  Data  Model  Using  STEP 

PDSS  Product  Definition  Standard  System 

PDU  Protocol  Data  Unit 

PEO  Program  Executive  Officer 

P&L  Production  and  Logistics 

PLA  Patent  Licensing  Agreement 

PNGV  Partnership  for  the  Next  Generation  Vehicle 

PM  Program  Manager 

PO  Program  Office 

POC  Point  of  Contact 

PPF  Platform  Proto-Federation 

PRIMES  Preflight  Integration  of  Munitions  and  Electronic  Systems 

Ri&D  Research  and  Development 

RAM  Random  Access  Memory 

RAM  Reliability,  Availability,  and  Maintainability 

RAMP  Rapid  Acquisition  of  Manufactured  Parts  (Navy  program) 

RASSP  Rapid  Prototyping  of  Application  Specific  Signal  Processors 

RCS  Radar  Cross  Section 

RDA  Research,  Development  and  Acquisition 

RDC  Research  and  Development  Center 

RD&E  Research,  Development  and  Engineering 

RDEC  Research,  Development  and  Engineering  Center 

RDT&E  Research,  Development,  Test  and  Engineering 

^  Radio  Frequency 

RFP  Request  for  Proposal 

ROI  Return  on  Investment 

RTTC  Redstone  Technical  Test  Center  (Army) 

SAIC  Science  Applications  International  Corporation 

SBA  Simulation  Based  Acquisition 

SBD  Simulation  Based  Design 

SC-2 1  Surface  Combatant  -  2 1  st  Century  (Navy  Program) 

SDRC  Structural  Dynamics  Research  Corporation 

SEP  Simulation  Endorsement  Process 

SIIRCM  Suite  of  Integrated  Infra-Red  Countermeasures 

SIL  System  Integration  Laboratory 

SIMCORE  Simulation  Common  Object  Repository  Environment 

SIMNET  Simulation  Network 

SOW  Statement  of  Work 


SPM  Smart  Product  Model 

SSA  Software  Support  Activity 

SSP  Simulation  Support  Plan 

STAF  Simulation  Test  Acceptance  Facility  (at  RTTC) 

STEP  International  Standard  for  the  Exchange  of  Product  Data 

STRICOM  Simulation,  Training  and  Integration  Command 
SWIL  Software  in  the  Loop 

T&E  Test  and  Evaluation 

TACOM  Tank  Automotive  Command  (Army) 

TACTICS  Tri-Service  Advanced  Countermeasures  and  Threats  Integrated  Combat 
Simulation 

TAMIP  Target  Acquisition  Model  Improvement  Program 

TARDEC  Tank  Automotive  Research,  Development  and  Engineering  Center 

TECOM  Test  and  Evaluation  Command  (Army) 

TEMA  Test  and  Evaluation  Management  Activity  (Army) 

TEMP  Test  and  Evaluation  Master  Plan 
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TILV  Target  Interaction,  Lethality  and  Vulnerability 
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TRP  Technology  Reinvestment  Program 

TVWS  Tracked  Vehicle  Work  Station 

USD  Under  Secretary  of  Defense 

VISION  Visual  Simulation  and  Organizational  Network  (Lockheed  Martin 
program) 

VM  Virtual  Manufacturing 

VNS  Virtual  Notional  Ship 

VP  Virtual  Prototyping 
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VSWE  Virtual  Ship  Warfare  Environment 
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V&V  Verification  and  Validation 
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WBS  Work  Breakdown  Structure 

WIP  Work  in  Progress 
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WWW  World  Wide  Web 
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capabilities  and  to  upgrade  M&S  based  on  field  test  results,  lessons  learned,  and  other  sources  of 
data.  The  study  stated  that  M&S  should  be  used  for  earlier  identification  of  system  problems.  It 
should  also  provide  a  connecting  link  between  phases  of  testing,  and  between  test  organizations. 


Major  barriers  to  the  increased  use  of  models  and  simulations,  in  addition  to  the 
technical  challenges  of  their  proper  use,  are  (1)  the  need  to  overcome  unfounded 
perceptions  regarding  their  credibility  and  realism,  and  (2)  that  their  use  would  increase 
T&E  cost. 

The  study  saw  a  need  for  better  policy  on  use  of  M&S  in  the  T&E  process,  in  part  to 
demonstrate  its  potential  for  improving  the  quality  and  reducing  the  resources  required  in  testing. 
This  includes  the  need  for  a  single  coordinating  point,  a  supporting  technology  base,  and 
appropriate  training  for  Army  professionals.  M&S  was  found  to  have  potential  as  a  unifying 
framework  for  operator,  logistics,  and  maintenance  functions.  Other  advantages  of  using  M&S 
are  listed  in  Appendix  B  of  the  study,  with  references. 

2.  Army  Science  Board,  “1991  Summer  Study:  Final  Report  on  Army  Simulation 
Strategy,”  December  1991,  Office  of  the  Assistant  Secretary  of  the  Army  (RDA). 

The  study  assessed  the  status  of  M&S  technology  with  regard  to:  barriers,  enhancement 
opportunities,  payoffs  and  benefits,  and  its  role  in  development  and  testing  and  in  RD&A.  The 
study  found  use  of  M&S  to  be  widespread  and  important,  and  stated  that  use  of  the  Electronic 
Battlefield  (EB),  as  defined  in  the  report  (see  figure  C-1  for  a  depiction  of  the  functions  and 
opportunities  of  the  EB),  can  revolutionize  the  Army’s  way  of  doing  business  while  reducing 
cost  and  increasing  quality. 
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Figure  C-1 :  Electronic  Battlefield  Functions  and  Opportunities 


Simulation  should  be  used  for  electronic  prototypes  through  all  phases  of  the  force 
development  and  materiel  acquisition  processes.  Use  of  early  electronic  prototyping  will  be 
invaluable  in  refining  requirements,  understanding  how  technology  will  fit  vidth  future  systems, 
and  help  evolve  physical  prototypes.  The  full  spectrum  of  capabilities  are  needed,  from  low 
resolution  to  high  resolution  and  hardware-in-the-loop.  “...  for  FDT&E  and  OT,  electronic 
prototypes  should  be  used  both  as  a  preview  of  all  possible  required  physical  tests  and  as  an 
extension  beyond  physical  testing  capability.”  This  will  reduce  the  need  for  physical  testing,  but 
will  in  turn  require  a  fundamental  change  in  policy  toward  testing,  one  which  will  probably  have 
to  evolve  over  time  as  the  EB  proves  itself. 

The  EB  should  serve  as  the  primary  test  environment  for  early  evaluation  of  operational 
utility.  The  quality  and  consistency  of  the  data  bases  and  models  for  the  EB  will  be  of 
paramount  importance.  Key  models  must  be  labeled  ‘certified’  to  show  they  are  acceptable, 
while  new  products  can  be  ‘not  certified’  for  introduction  of  new  concepts  for  testing.  Army 
resources  must  focus  on  specific  Army  needs,  leaving  R&D  for  underlying  technology  to  others 
such  as  DARPA  and  the  private  sector. 

3.  IDA,  “Army  Aviation  Simulation  Survey,”  July  1992. 

Conducted  during  a  period  of  rapid  technological  change,  this  survey  helped  determine 
priorities  for  new  capabilities  in  Army  Aviation  RD&A  and  related  areas  such  as  training.  The 
report  also  describes  the  need  for  a  more  unified  effort  with  regard  to  aviation  related  M&S.  The 
report  takes  a  broad  view  of  the  integration  of  M&S,  including  live  forces  on  instrumented 
ranges  and  virtual  and  constructive  war  games.  A  large  number  of  the  survey 
responses/recommendations  from  1992  have  been  acted  upon  already.  The  following  areas 
pertinent  to  acquisition  M&S  (as  well  as  for  training  applications  in  many  cases)  were  cited  as 
needing  more  attention: 

•  greater  fidelity  in  battlefield  representation, 

•  better  networking  of  simulations/simulators, 

•  reconfigurable  cockpits  and  mission  equipment  packages  (MEPs), 

•  better  ability  to  evaluate  subsystems/MEPs  relative  to  the  total  system, 

•  faster,  smaller  computational  systems, 

•  better  automated  tools  for  building  models,  and 

•  ability  to  interface  between  high  and  low  resolution  simulators. 

This  survey  was  conducted  at  the  time  that  technological  advances  were  beginning  to 
permit  use  of  distributed  interactive  simulations  capable  of  man-in-the-loop  warfighting  on 
futuristic  battlefields.  The  report  is  intended  to  help  determine  priorities  on  where  money  should 
be  spent  for  new  and  enhanced  capabilities  in  the  areas  of  conceptual  design,  performance 
analysis,  testing,  manufacturing  process  design  and  production,  and  training  and  mission 
rehearsal  from  crew  to  joint  and  coalition  levels.  Ten  organizations  were  surveyed  with  the 
Aviation  Center  and  the  Aviation  Systems  Command  submitting  multiple  reports.  The  report 


summarizes  the  capabilities  of  Army  simulation  facilities  relevant  to  Army  aviation  R«&;D,  as 
well  as  major  defense  industry  firms  active  in  aviation  R&D. 

4.  Rand  (National  Defense  Research  Institute),  “Enhancing  Weapon  System  Analysis: 
Issues  and  Procedures  for  Integrating  an  R&D  Simulator  with  a  Distributed  Simulation 
Network,”  1994. 

A  1994  Rand  study  describes  an  important  limitation  of  the  current  process  —  the 
inability  to  evaluate  high  fidelity  representations  of  developing  systems  in  a  realistic  overall 
combat  environment.  It  offers  a  proposal  for  using  simulation  and  networking  to  evaluate 
systems  and  resolve  deficiencies  in  current  weapons  system  analysis  methods.  The  proposal  is  to 
combine  high  fidelity  representations  of  developing  systems  with  the  combat  simulation 
environment  of  SIMNET.  In  this  way,  it  is  possible  to: 

•  simulate  a  system’s  specifications/capabilities  in  sufficient  detail  to  measure  performance 
differences  in  alternative  approaches  or  technologies, 

•  simulate  the  system’s  performance  in  the  military  unit  configuration  and  operational 
environment  for  which  it  is  being  developed,  and 

•  have  humans  operating  the  system  in  the  intended  combat  environment  and  making  real-time 
decision. 

Such  an  approach  can  overcome  the  drawbacks  associated  with  use  of  professional 
judgment,  combat  simulation,  R&D  human-operated  simulators,  and  prototypes  in  live  T&E. 
Each  of  these  tools  are  currently  used  in  the  traditional  process,  and  are  useful,  but  are  not  able  to 
achieve  the  combined  abilities  outlined  in  the  bullets  above. 

5.  Army  Science  Board,  “1993  Summer  Study:  Final  Report  on  Innovative  Acquisition 
Strategies  for  the  90s,”  July  1994,  Office  of  the  Assistant  Secretary  of  the  Army  (RDA). 

This  study  makes  five  key  recommendations.  The  first  pertains  to  technology  upgrading 
through  use  of  horizontal  technology  integration  (HTI)  and  vertical  technology  insertion  (VTI) 
for  a  few  key  programs.  This  will  help  save  constrained  resources.  The  second  pertains  to 
digitization  (and  the  need  to  establish  priorities,  codify  operational  requirements,  and  enforce 
standards  to  create  an  integrated  system).  The  third  is  exploitation  of  simulation  by  better 
upgrades  and  new  starts  and  by  exploiting  technology  developed  elsewhere  and  adapted  to  Army 
needs.  The  fourth  is  acquisition  reform  including  use  of  the  2-step  acquisition  process  for  HTI 
and  VTI.  The  fifth  pertains  to  resourcing  and  the  need  to  reduce  acquisition  and  fixed  costs  in 
order  to  save  modernization  dollars.  Figure  C-2  is  an  overview  of  the  evolving  environment. 
These  pressures  will  cause  emphasis  to  be  placed  on  upgrades  in  the  future. 
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Figure  C-2:  Evolving  Acquisition  Environment 

A  centerpiece  of  the  study  is  a  set  of  recommendations  for  projects  which  are  suitable  for 
HTI  and  VTI.  The  simplified  2-step  model  shown  in  figure  C-3  is  related  to  future  application  of 
technology  integration.  The  point  is  that  technical  risk  must  be  identified  and  mitigated  early 
through  open  interaction  and  teamwork  between  Government  and  contractors. 


Current  Model: 

MSO 


Determination! 
I  of  Mission  i 


I  Need 


j  Concept 
Exploration 
I  &  Definition 


Demonstration 

& 

Validation 


-  Early  Selection  of  Preferred  Solution 

-  Technology  Risks  Often  Passed  to  EMD 


Phase  II 

Engineering  & 
Manufacturing 
Development 


Phase  III 

Phase  rv 

Production  ^ 

S.  Operation 

&  Deployment 

&  Support 

-  Many  Long  Duration  Expensive  Programs; 
Stretched  Production  Runs;  Changes 


Simplified  2-Step  Model: 


^  Determination 
I  of  Mission 
I  Need 


PM  Risk  Matrix 
Available 

\  Interactive 

J  R&D 


Start  EMO/Production 

Integrated  \  \ 
Engineering  &  \  \ 
Production  (lE&P)  \  \ 


-  Focused  Technology  Maturation 
Through  Development  of  Options 

-  Concurrent  Engineering 

-  Battle  Lab  Simulation 

-  Demonstrated  Technology  Risk 
Reduction 


-  USD(A)  Review/Risk  Matrix 

-  Few  Short  Duration,  Short  Production 
Run,  Low  Risk  Programs 

-  HTI  and  VTI  of  Proven  Technology 


Figure  C-3:  A  2-Step  Strategy  for  Technology  Upgrades 


Most  important  is  an  understanding  that  the  single  most  frequent  reason  for 
program  failure  is  the  application  of  immature  technology.  This  is  entirely  consistent 
with  past  studies,  and  in  particular,  with  the  ‘Betti’  acquisition  study  where  an  empirical 
data  base  of  several  hundred  programs  clearly  identified  technological  risk  as  the  single 
most  important  cause  of  program  failure. 

The  1993  Summer  Study  describes  why  the  2-step  process  is  especially  applicable  to 
technology  upgrades/HTWTI  programs  in  terms  of  risk  management.  The  2-step  process 
requires  a  cultural  change  in  terms  of  trust  and  openness.  Such  a  quality/cycle-time  driven 
culture  can  help  ensure  successful  programs. 

The  entire  focus  is  proving,  via  demonstrations,  that  the  employed  technology  will 
perform  to  the  necessary  levels  by  the  time  R&D  (the  first  step)  is  finished.  The 
Government-Industry  team  does  this  with  the  full  knowledge  that:  at  the  end  of  step  one, 
the  technology  performance  must  be  met  or  the  program  will  be  terminated,  and  that 
moving  to  step  2  (actual  integrated 

engineering  and  manufacturing  development)  with  any  technology  risk  still  existing  is 
unacceptable. 


The  study  goes  on  to  deseribe  risk  reduction  and  tools  for  managing  risk.  Simulation 
must  be  used  early  to  reduce  risk.  It  also  discusses  the  need  for  cutting  unneeded  specifications 
and  standards,  better  allocation  mechanisms  for  funding,  and  use  of  the  2-step  process.  Major 
shifts  in  the  culture  and  managerial  paradigm  are  needed. 

This  study  concurred  with  the  1991  Army  Science  Board  study  finding  that  the  Army 
should  exploit  M&S  technology  that  is  developed  elsewhere  (i.e.,  take  advantage  of  the 
commercial  simulation  revolution)  and  not  invest  except  in  areas  of  Army-unique  needs.  Much 
of  the  report  pertained  to  improved  training  systems.  The  key  point  relating  to  M&S  is  to 
“develop  and  implement  a  common  architecture  and  associated  standards  which  support  the 
interpretability  and  interconnectivity  functions  and  dynamic  interactions  which  affect  the  conduct 
and  results  of  warfighting.” 

Much  of  the  information  from  the  1993  Summer  Study  appears  valid  in  terms  of  the 
discussions  conducted  during  the  course  of  the  current  study.  Integration  of  technology,  or 
gradual  insertion  of  upgrades,  is  becoming  increasingly  prevalent  for  both  existing  and  new 
programs.  This  goes  hand-in-hand  with  increased  attention  to  greater  risk  reduction,  shorter 
cycle  times  and  faster  insertion  of  incremental  improvements,  and  better  integration  of  new 
capabilities  across  the  force.  The  2-step  strategy  more  closely  resembles  evolving  paradigms  for 
both  upgrades  and  new  start  programs  as  it  requires  closer  team  integration  and  reduction  in 
barriers. 


6.  Acquisition  Task  Force  on  M&S,  “Report  Out  of  Acquisition  Task  Force  on  M&S,” 
June  1994. 

The  Acquisition  Task  Force  on  M&S  (ATFMS)  interviewed  many  program  managers  and 
supporting  personnel  and  ftinctional  experts  and  concluded  that  much  system  and  program 
analysis  is  conducted  in  isolation,  primarily  due  to  lack  of  resources,  infrastructure,  and 
direction.  However,  the  trend  is  toward  greater  integrated  functional  analysis.  This  generally 
encompasses  “analysis  that  considers  the  impact  of  decisions,  trade-offs,  and  risks  both  within 
and  across  acquisition  functional  disciplines.”  Another  conclusion  was  that,  due  to  budget 
reductions,  program  offices  are  making  much  greater  use  of  M&S  to  reduce  system  development 
cost.  Others  factors  causing  increased  use  of  M&S  are  the  increased  demands  for  program 
information,  and  increased  capability  and  availability  of  M&S  and  related  interconnectivity  tools. 

The  ATFMS  was  established  by  Dr.  Anita  Jones  to  recommend  actions  to  more 
effectively  integrate  the  use  of  M&S  throughout  the  acquisition  process.  The  ATFMS  found 
many  examples  of  progress  toward  exploiting  the  potential  of  M&S.  Programs  specifically 
mentioned  include  the  Air  Force  F-22,  Army  Comanche,  and  Navy  Submarine  Off-Board  Mine 
Search  System  programs. 

There  were  examples  of  M&S  integration  initiatives  across  functional  areas.  The 
DARPA  DICE  (DARPA  Initiative  on  Concurrent  Engineering)  and  MADE  (Manufacturing 
Automation  and  Design  Engineering)  programs,  using  the  Agile  Manufacturing  Information 


Infrastructure,  are  attempting  to  integrate  manufacturing  enterprise  disciplines.  The  DARPA 
program  for  Simulation  Based  Design  is  developing  technology  to  support  the  transition  of 
design  efforts  from  the  current  use  of  drawings  and  mockups  to  use  of  virtual  prototypes.  Initial 
efforts  in  ship  acquisition  will  later  transition  to  interfacing  this  application  with  cost, 
manufacturing,  and  logistics  M&S  applications.  The  services  and  DoD  agencies  have  developed 
joint  standard  applications  and  utilities,  such  as  Joint  M&S  System  (JMASS),  Joint  Simulation 
System  (JSIMS),  Distributed  Interactive  Simulation  (DIS),  and  Defense  Simulation  Internet 
(DSI). 


Other  initiatives  are  integrating  modeling  vertically  at  specific  centers  and  labs,  for 
performance  and  effectiveness  analysis,  such  as  combining  Extended  Air  Defense  Simulation 
(EADSIM)  with  TAG  BRAWLER  to  support  Army  and  Air  Force  air  defense  and  air  operations 
centers  and  labs.  Another  example  is  the  Army’s  intra-  and  inter-fimctional  integration  for  the 
Advanced  Warfighting  Demonstrations  and  for  various  functions  such  as  the  Extended  Air 
Defense  and  Missile  Test  Bed. 

Promising  cases  of  infrastructure  investment  and  development.  Service  leadership  in 
centralizing  coordination  and  support  for  M&S,  management  innovation  and  development  of 
educational  programs  and  materials  were  cited.  On  the  negative  side,  the  ATFMS, 

discovered  problems  that  limit  the  extent  of  M&S  integration  in  a  reformed  acquisition 
process  and  impede  the  development  of  the  core  M&S  technologies  needed  to  facilitate 
such  reform.  The  most  important  limitations  are  in  the  areas  of  cross-functional 
architectures,  M&S  support  infrastructures,  M&S  management,  M&S  education  and 
training,  and  M&S  resourcing  practices. 

With  regard  to  cross-functional  architectures,  the  accelerating  need  for  acquisition  M&S 
caused  the  evolution  of  architectures  “that  are  inconsistent  with  architectures  evolving  in  the 
operations,  training,  and  requirements  analysis  community.”  The  other  commimities  were 
building  integrated  M&S  systems  with  common  environments  and  objects.  Organized  effort  is 
lacking  at  OSD  level  to  create  appropriate  technical  and  managerial  links  between  the  battlefield 
and  acquisition  environments. 

In  terms  of  M&S  support  infrastructure,  the  absence  of  an  overall  architecture  prevents 
the  optimum  return  on  investments.  Most  applications  are  unique  to  a  specific  program, 
developed  to  address  program-unique  problems,  and  are  often  contractor-unique  solutions  that 
the  government  doesn’t  own.  When  surveyed,  program  management  personnel  stated  they 
wanted  to  make  greater  use  of  M&S  but,  “wondered  about  the  nature  of  the  architecture  in  which 
their  simulations  might  be  used  and  with  which  they  should  be  compatible.”  In  addition. 

Since  PMs  use  system  development  funds  to  build  and  improve  models,  there  is 
little  incentive  to  use  common  standards  or  to  adopt  a  common  architecture  except  in 
instances  where  it  is  obvious  resources  can  be  saved  for  the  program.  Furthermore,  there 
is  little  incentive  to  integrate  models  across  functional  areas  because  programs  are 


managed  in  narrow  functional  stovepipes  at  the  center,  major  command,  Services,  and 
OSD  level. 

In  the  area  of  M&S  resourcing  practices,  the  report  found  that  decentralized  management 
resulted  in  inefficiencies.  As  discussed  imder  infrastructure,  functional  stovepiping  and  the  lack 
of  incentives  for  integration  results  in  models  that  are  not  properly  shared  and  duplicative.  Many 
respondents  believed  that  “...  one  activity  should  be  given  responsibility  for  data  maintenance 
and  limited  control  of  M&S  development  for  specific  environments,”  especially  for  configuration 
and  data  management  functions.  It  appears  that  the  USAF  has  best  centralized  configuration 
control  and  interface  management  control.  With  regard  to  M&S  management,  most  respondents 
wanted  to  see  an  identifiable  office  or  agency  at  OSD  level  with  responsibility  for  a  common 
acquisition  M&S  infrastructure. 

The  ATFMS  found  that  most  users  wanted  the  freedom  to  develop,  improve,  and 
integrate  their  models,  but  at  the  same  time,  they  do  not  want  to  reproduce  the  same 
incompatibility  problems  present  in  their  current  set  of  legacy  models.  Many  offices 
believe  that  an  identifiable  office  at  the  OSD  level  needs  to  provide  an  overall  vision  and 
coordinate  evolution  of  the  M&S  acquisition  architecture. 

The  final  area  was  the  need  for  more  education  and  training  to  inform  prospective  users  of 
available  M&S  products.  Potential  users  need  to  know  what  models  are  available  to  provide  the 
right  answers  in  the  proper  timeframe  and  at  an  acceptable  price.  Many  of  the  proposed 
recommendations  proved  to  be  organizational  rather  than  technical.  Three  principle 
recommendations  were  offered  by  the  ATFMS  to  more  effectively  integrate  the  use  of  M&S 
throughout  the  acquisition  process.  The  first  involved  the  establishment  of  a  JPO  (joint  program 
office)  for  M&S  to  help  promote  the  establishment  of  the  desired  M&S  infrastructure.  Second,  is 
that  DDR&E  establish  an  adequately  resourced  M&S  staff  in  ODDR&E  to  provide  the  support 
needed  for  its  M&S  agenda.  Finally,  that  DDR&E  should  sponsor  experiments  in  M&S 
technologies  in  real  programs  so  the  Services  can  gain  greater  knowledge  of  their  utility, 
benefits,  costs  and  so  forth. 

With  regard  to  the  last  recommendation,  the  report  proposes  criteria  that  can  be  used  for 
selecting  appropriate  programs  for  M&S  experimentation.  Among  the  criteria  are:  phase  of 
acquisition,  status  of  requirement,  level  of  funding,  use  of  integrated  decision  team  methods, 
program  size,  program  experience  in  use  of  M&S,  infrastructure  for  M&S,  and  applicability  of 
results  to  other  DoD  acquisition  programs. 

A  series  of  metrics  are  proposed  for  use  in  M&S  experiments.  Metrics  were  selected 
based  on  their  ability  to  accurately  measure  the  performance  of  the  system  under  experiment,  to 
provide  proper  focus  for  experiment  managers,  and  to  provide  “...useful  indicators  for  senior 
management  to  judge  the  overall  performance  of  the  experiment  and  its  merits  in  other  program 
environments...”  The  metrics,  which  relate  to  the  present  study,  are  summarized  below. 

COST  SAVINGS: 

-  Extent  to  which  estimated  production  unit  cost  can  be  reduced 


-  Extent  to  which  system  operation  cost  can  be  reduced 

-  Extent  to  which  program  can  achieve  savings  in  design  and  development  tasks,  relative 
to  cost  estimates  based  on  current  methods 

TIME  SAVINGS: 

-  Extent  to  which  the  number  of  iterative  cycles  has  increased  while  reducing  the  total 
time  spent  on  those  cycles 

-  Reduction  in  time  required  to  implement  all  supporting  aspects  of  design  (e.g., 

time  required  to  change  management  documentation) 

-  Extent  to  which  estimated  time  for  the  EMD  phase  is  reduced 

QUALITY  IMPROVEMENTS: 

-  Quality  of  design  reviews  based  on  improved  methods  of  visualizing  data 

-  Quality  of  design  information  passed  to  and  from  industry 

RISK  MITIGATION: 

-  Extent  to  which  risks  associated  with  component  design  or  selection  can  be 

mitigated 

-  Extent  to  which  risks  associated  with  use  of  individual  processes  and  combinations 

of  processes  can  be  anticipated  and  appropriately  compensated 

-  Extent  to  which  M&S  can  be  employed  to  make  the  T&E  process  more  effective  and 
efficient 

-  Extent  to  which  ‘system  of  systems’  issues  can  be  addressed  during  the  conceptual 
design  phase 

-  Extent  to  which  M&S  can  be  employed  to  identify  and  address  critical  support  issues 
early  in  a  program 

The  ATFMS  report  was  published  in  June  of  1994,  and  many  of  its  recommendations 
have  been  adopted  in  some  form.  Nonetheless,  many  of  the  issues,  terms  of  reference,  and 
conclusions  are  still  applicable  for  this  study.  The  concerns  and  realities  outlined  in  the  study, 
such  as  declining  budgets,  increasing  emphasis  on  acquisition  reform,  enabling  technologies,  and 
more  dual-use  applications,  have  caused  DoD  to  transition  to  a  more  effective  and  efficient 
acquisition  process.  Some  of  the  Service  initiatives  in  response  to  the  problems  included  the 
Army  requirement  that  all  ASARC  programs  have  a  Simulation  Support  Plan  (SSP),  the  Navy 
N80  initiative  to  assess  how  M&S  can  lead  to  a  better  acquisition  system,  and  Air  Force 
regulations  requiring  manufacturing  simulation  plans.  Many  of  these  efforts  focus  on 
development  of  an  integrated  acquisition  environment.  The  ATFMS  summarized  M&S  needs 
using  the  standard  categories  for  M&S  functional  areas  as  seen  in  figure  C-4. 


T&E  Analysis  P&L  R&D  ETMO 


1.  Standards  &  Commonality  X 

2.  Data  Bases  X 

3.  Models  and  Simulations  X 

4.  Scenarios 

5.  Tools 

6.  Information  Sharing  X 

7.  Interoperability  X 

8.  Networks  X 

9.  VV&A  X 

10.  Multi-level  Security  X 

1 1 .  M&S  Policy  Statements  X 

12.  Education  of  Managers  X 

13.  Education  of  Practitioners  X 

14.  Environmental  Representations  (X) 

15.  Behavioral  Representations 


16.  M&S  Development  Environment  X 

17.  New  Models  &  Simulations 


X 

X 

X 


X 


X 

X 

X 

X 


X  X 
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X  X 
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Figure  C-4:  Summary  of  M&S  Needs 


7.  IDA  Paper  P-3062,  “Review  of  Defense  M&S  Office  Projects  and  Activities  for  FY 
92  to  94,”  August  1995,  Institute  for  Defense  Analysis,  M.H.  Hammond,  et.  al. 

This  document  reviews  and  provides  summary  information  on  48  projects  and  17 
activities  conducted  from  FY92  to  FY94.  The  projects  and  activities  are  related  to  1  of  9 
technical  areas  (as  might  be  expected,  the  areas  receiving  the  most  emphasis  are  architecture, 
information/databases,  environmental  representation,  and  interoperability  with  C3I  systems)  and 
5  functional  areas  (ETMO  received  the  greatest  emphasis).  Some  of  the  projects  of  interest  to 
this  study,  and  which  are  discussed  more  fully  later,  include: 

FY92  Projects:  Joint  M&S  System  (J-MASS)  -  Department  of  the  Air  Force  and 
J-MASS  PO  —  Architecture  and  methodology  for  integrating  legacy  data  and  tools,  also  for 
scenario  development  and  for  T&E. 

Simulation  Based  Design  for  Military  Systems  Supportability  and  Human  Factors 
—  DARPA  and  University  of  Iowa  Center  for  Simulation  and  Design  Optimization  —  Extend 
SBD  to  ground  vehicles  to  include  maintainability  aspects. 

FY93  Projects:  Integrated  Radar  and  Infrared  Analysis  and  Modeling  (IRIAM)  — 
DoN  and  Naval  Air  Warfare  Center  —  Develop  a  prototype  for  improved  analysis  of  multi- 
spectral  measures  and  weapon  systems  data. 


Prediction  of  Electronic  Equipment  Reliability  Using  Computer  M&S  ~  DA  and 
AMS  AA  -Develop  improved  models  of  electronic  system  reliability  using  physics-of-failure 
concepts. 


FY94  Projects:  AAAV:  A  Pilot  M&S  Program  to  Streamline  T&E  within  the 
Acquisition  Process  -  DoN,  PM  for  AAAV  -  Analyze  M&S/DIS  resources  that  can  be  applied 
to  reducing  acquisition  cost  and  increasing  total  system  effectiveness. 

The  Integrated  Priority  List  (IPL)  process  was  used  to  select  and  prioritize  the  projects. 
Tables  in  the  report  list  each  project  by  priority,  briefly  describe  it,  and  show  which  categories  it 
impacts,  either  primarily  or  secondarily.  Section  4  discusses  how  the  projects  and  activities 
contributed  in  three  areas  of  technological  interest,  including  the  area  of  advances  in  synthetic 
environments. 


These  advances  resulted  from  efforts  to  improve  representation  of  the  physical 
environment ,  to  improve  representation  of  weapons  systems  and  military  units,  and  to 
define,  standardize,  and  enhance  a  number  of  simulation-related  communication 
protocols. 

Much  of  the  report  is  devoted  to  discussion  of  standardization  and  enhancement  of 
simulation  protocols.  The  IDA  Paper  also  describes  the  organization  and  initiatives  of  DMSO, 
including  publications  and  regulations  for  which  it  is  responsible,  enhancement  of  simulation 
protocols,  and  efforts  under  the  Defense  M&S  Initiative. 


8.  M&S  Benefits  Task  Force,  “Report  Out  to  the  M&S  Benefits  Task  Force,”  December 
5, 1995. 

The  goal  of  the  M&S  Benefits  Task  Force  (MSBTF),  authorized  by  the  M&S  Working 
Group  under  DMSO,  was  to  capture  documented  reports  of  quantifiable  M&S  benefits.  The 
MSBTF  collected  data  using  computerized  requests  for  information  and  by  other  means.  Data 
was  collected  in  the  application  areas  of  acquisition,  training,  and  analysis. 

Very  limited  responses  were  received  by  the  MSBTF  from  two  data  calls,  which  limited 
the  ability  of  the  task  force  to  conduct  a  formal  assessment.  The  report  found  that  there  is  no 
formal  reporting  mechanism  for  reporting  benefit  information,  nor  is  there  a  process  for 
objectively  assessing  gathered  information.  The  report  concluded  that  analytic  frameworks  need 
to  be  developed  and  applied  throughout  DoD  in  order  to  imderstand  the  benefits  of  M&S. 

In  the  area  of  target  interaction,  lethality,  and  vulnerability  (TILV),  a  review  of  21  case 
studies  showed  an  average  30  to  1  ROI.  The  Army’s  Missile  Command  reported  over  $320M  in 
cost  avoidance  and  savings  from  10  case  studies.  The  Army’s  TECOM  reported  on  8  systems  in 
which  cost  avoidance  totaled  about  $80M  using  the  Virtual  Proving  Ground  concept.  Finally, 
the  Apache  Longbow  program  conducted  two  similar  test  events,  one  using  extensive  simulation 
and  the  other  using  physical  equipment  and  personnel.  “The  simulation-supported  event 


executed  twice  as  many  trials,  with  fewer  personnel,  in  less  time,  at  lower  risk  to  personnel,  for 
$700,000  versus  $4  million.” 

Two  members  of  the  task  force  prepared  analyses  to  provide  measures  of  effectiveness 
and  methods  of  analysis  (Appendices  A  and  B  of  the  cited  report).  Appendix  A  describes 
possible  reasons  why  the  value  of  using  M&S  is  difficult  to  quantify:  lack  of  accepted  metrics, 
difficulty  in  obtaining  supporting  data,  and  difficulty  in  identifying  a  valid  baseline  from  which 
to  measure  increases  in  value.  M&S  capabilities  are  divided  into  the  two  broad  areas  of  technical 
and  functional.  The  technical  area  deals  with  mechanisms  that  make  the  M&S  application  work. 
The  functional  area  considers  how  M&S  applications  will  be  used.  Tables  C-2  and  C-3  provide  a 
sample  of  important  area  topics  and  proposed  quantitative  and  qualitative  metrics  or  MOEs  for 
each. 


Appendix  B  compares  alternative  methods  of  calculating  cost  effectiveness  using  three 
case  studies  pertaining  to  M&S  use.  Analyses  results  are  as  follows: 

There  are  four  basic  categories  of  effectiveness  measures  obtained  from  applying 
M&S  -  doing  it  better,  doing  it  faster,  doing  it  cheaper,  and  doing  it  at  all.  However,  the 
metrics  are  often  hard  to  measure  especially  in  terms  of  dollars.  What  value  is  ascribed  to 
a  safer  or  better  process?  For  example,  what  is  the  value  of  avoiding  a  disaster,  near 
miss,  or  a  high-risk  test?  What  is  the  value  of  an  event  or  a  series  of  thousands  of  discrete 
events  that  would  not  be  possible  without  use  of  a  simulation?  What  is  the  value  when  a 
faster  and  more  eost  efficient  solution  is  foxmd  by  using  M&S? 


TABLE  C-2:  Technical  Areas 


•  Architecture:  the  high-level  system  and  software  design  of  the  M&S  tool.  A  quantitative  metric  is  the  amount 
or  percent  of  legacy  migration  to  the  new  system. 

•  Environmental  representation:  how  well  the  real  world  is  portrayed  in  the  synthetic  environment.  A  candidate 
quantitative  metric  is  stimulation,  or  the  number  of  environmental  effects  that  the  M&S  application  can  portray. 
A  possible  qualitative  metric  is  immersion,  or  how  real  the  environment  feels  to  a  human. 

•  Information/database:  methods  for  M&S  tools  to  store  or  access  information  or  for  data  modeling.  A 
quantitative  metric  is  level  of  effort  required for  collaboration  and  reuse, 

•  Interoperability:  how  various  M&S  tools  interface  and  operate  together.  A  qualitative  metric  proposed  for  this 
area  is  called  level  playing  field.  This  embodies  the  idea  of  a  neutral  evaluation  tool  that  treats  all  Service  roles 
and  missions  fairly. 

•  Networking:  how  data  is  shared  between  dispersed  and  remote  M&S  tools.  Possible  quantitative  metrics  are 
cost  per  megabit  per  second  and  latency.  A  possible  qualitative  metric  is  immersion,  the  quality  of  realistic 
sensory  stimulation. 

•  VV&A:  the  process  of  fully  approving  M&S  tools.  A  quantitative  metric  is  cost  avoidance  for  VV&A 
procedures  that  enhance  reuse.  A  proposed  qualitative  metric  is  enhanced  decision  support,  such  that  the 
YYScA  process  results  are  ‘more  valid’  and  lead  to  better  quality  decisions. 

•  Instrumentation:  the  details  of  the  infrastructure  that  help  send  state  variables  between  M&S  tools.  A  proposed 
quantitative  metric  is  risk  reduction,  in  that  the  instrumentation  helps  in  position  reporting  and  similar  actions 
that  can  help  reduce  risk.  A  qualitative  metric  is  merger  of  C4I  with  M&S.  For  a  number  of  reasons  (cost, 
simplicity,  integration),  it  is  desirable  for  these  two  disciplines  to  merge. 


TABLE  C~3:  Functional  Areas 

•  Analysis:  use  of  M&S  to  conduct  experiments  to  extract  useful  data.  A  candidate  quantitative  metric  is  net 
utility  defined  as  how  the  M&S  tools  allow  an  analyst  to  better  understand  a  process  or  to  gain  useful  data.  A 
qualitative  metric  is  data  quality  or  the  level  to  which  the  obtained  data  is  useful  for  analysis. 

•  Acquisition  (R&D):  M&S  used  to  enhance  concept  evaluation,  prototyping,  and  so  forth.  Possible  quantitative 
metrics  are  cost  savings,  and  number  of  options  considered.  Possible  qualitative  metrics  are  brainstorming  and 
unique  capability.  The  first  pertains  to  the  ability  of  M&S  to  explore  a  wider  array  of  options  before  deciding 
on  a  solution  or  approach.  The  latter  applies  to  actions  that  are  difficult  to  do  in  real  life  (i.e.,  without  M&S). 

•  Acquisition  (T&E):  conducting  developmental  and  operational  testing  using  M&S.  Use  of  cost  avoidance  as  a 
quantitative  metric  is  well  supported  by  a  number  of  studies.  A  number  of  qualitative  metrics  are  proposed 
including  test  planning,  development  of  MOPs  and  development  of  MOEs.  Tliese  qualitative  metrics  allow 
users  to  describe  how  well  M&S  tools  assist  with  the  stated  tasks. 

•  Acquisition  (P&L):  use  of  M&S  for  functions  such  as  manufacturing,  process  analysis,  and  support  planning. 
Possible  quantitative  metrics  are  cost  savings  resulting  from  simulation  that  assists  in  problem 
identification/correction,  and  number  of  options  considered,  in  terms  of  options  using  production  equipment 
and  constraints. 


Table  C-4  provides  useful  guidance  on  how  to  determine  the  value  of  M&S  tools  as  they 
are  used  for  various  aspects  of  the  acquisition  process. 


TABLE  C-4:  Value  of  M&S  Tools  for  Various  Parts  of  the  Acquisition  Process 


MEASURE  OF 

EFFECTIVENESS  (MOE) 

DETERMINATION  OF  MOE 

RDT&E  AND  ANALYSIS: 

FASTER 

Better  adherence  to  schedule 

Review  daily  expenditures 

Use  of  virtual  prototyping 

Look  at  turn-around  time  from 
physical  models 

BETTER 

Value  of  more  detailed  analysis 

Reduced  number  of  design,  S/W, 
planning  changes 

Value  added  of  considering  more 
alternatives 

Can  review  P3I  and  other 
alternatives 

Value  added  of  making  better 
decisions 

Reduced  false  starts/backup  plans 
for  risk  mitigation 

CHEAPER 

Cost  savings  of  using  new 
methods 

Better  review  of  current  and 
projected  cost;  “what  ifs” 

AT  ALL 

Value  added  of ‘‘executable” 
requirements 

Costs  of  erroneous  requirements: 
redesigns,  ambiguity 

BIDT&E,  DESIGN  PHASE: 


FASTER 

Reduction  of  design  iterations 

Compare  to  similar  effort 

Automatic  design  documentation 

Compare  to  manual  methods 

BETTER 

Incorporate  maintenance, 
logistics,  and  production 
considerations 

Estimates  of  reduced  LCC  from 
what  simulations  reveal 

CHEAPER 

Use  of  virtual  prototypes 

Cost  of  physical  models 

AT  ALL 

Evaluation  of  designs  under  more 
situations 

Estimated  costs  of  design  failure 
under  such  situations 

TEST  AND  EVALUATION: 


FASTER 

Better  adherence  to  schedule 

Daily  cost  of  ranges,  program 
slips 

Better  use  of  flight  test  time 

Percent  of  test  time  wasted 

BETTER 

Value  added  of  “monte  carloing” 
test  conditions 

Percent  of  ops  requirements  not 
physically  tested,  but  inferred 
from  testing 

Value  added  of  rehearsing  test 

Percent  of  tests  wasted 

Earlier  identification  of  problems 

Look  at  cost/spending  curve  by 
phase  of  project;  look  at  cost  of 
ECPs  by  phase 

CHEAPER 

Use  of  virtual  prototyping 

Cost  of  physical  models 

AT  ALL 

Evaluations  of  designs  under 
more  situations 

Estimated  costs  of  design  failure 
under  such  situations 

